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ABSTRACT 

Simulation  is  widely  used  within  the  Australian  Army  and  other  defence  forces  for 
marksmanship  training.  However,  there  is  a  lack  of  data  that  quantifies  the  training 
effectiveness  of  these  devices,  and  informs  the  appropriate  mix  of  live-fire  and  simulation. 
This  report  documents  a  study  which  measured  the  effect  of  additional  simulator  training  on 
a  basic  live-fire  qualification  task  with  the  M4  weapon.  The  study  was  conducted  with  thirty- 
six  Australian  Defence  Force  personnel  enrolled  in  an  Army  training  course.  Following  initial 
marksmanship  training,  one  group  (n=18)  completed  additional  simulator  training  while  the 
other  group  (n=18)  proceeded  straight  to  live-fire  qualification.  The  group  that  received 
additional  training  in  the  simulator  achieved  slightly  faster  live-fire  qualification;  however 
trainees  in  both  groups  required  numerous  attempts  to  pass.  The  findings  show  that  the 
simulator  training  had  a  small,  positive  benefit,  but  that  the  overall  training  was  not 
particularly  satisfactory.  The  report  outlines  reasons  for  these  findings  and  provides 
recommendations  for  future  studies  that  could  explore  more  effective  training  methodologies. 
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Evaluation  of  the  Effectiveness  of  Simulation  for  M4 

Marksmanship  Training 

Executive  Summary 

Simulation  is  widely  used  within  the  Australian  Army  and  other  defence  forces  for 
marksmanship  training.  The  cost  of  live  ammunition,  the  availability  and  accessibility 
of  live-fire  ranges,  and  administrative  overheads  associated  with  planning  and 
conducting  live-fire  practices  all  make  the  use  of  marksmanship  simulators  an 
attractive  training  option.  Anecdotally,  it  is  believed  that  the  use  of  marksmanship 
simulation  leads  to  savings  in  overall  training  time  and  improved  live-fire 
performance;  however  there  is  little  empirical  data  to  support  this  belief.  Furthermore, 
within  the  Australian  Defence  Force  (ADF),  there  is  a  lack  of  data  to  inform  the 
appropriate  mix  of  simulator  and  live-fire  training  needed  to  achieve  specific  training 
objectives. 

This  report  documents  the  outcomes  of  a  study  examining  the  use  of  the  Weapons 
Training  Simulation  System  (WTSS)  for  basic  marksmanship  training  on  an  Australian 
Army  training  course.  The  aim  was  to  measure  the  effect  of  additional  simulator 
training  on  subsequent  live-fire  qualification.  Furthermore,  it  was  anticipated  that  the 
study  would  provide  Army  with  objective  data  to  inform  the  use  of  the  WTSS  for  basic 
marksmanship  training.  The  study  was  conducted  by  DSTO  Land  Operations  Division 
under  Task  ARM  07/163:  Training,  Learning  and  Performance. 

Thirty  six  ADF  personnel  enrolled  on  the  course  took  part  in  the  study.  As  part  of  the 
course,  trainees  were  required  to  pass  a  qualification  task  using  the  M4  weapon  with 
iron  sights  on  the  live-fire  range.  After  all  personnel  conducted  initial  marksmanship 
training,  one  group  conducted  the  qualification  practice  three  times  in  the  simulator 
prior  to  conducting  three  qualification  attempts  on  the  live-fire  range;  the  second 
group  conducted  three  qualification  attempts  on  the  live-fire  range  with  no  prior 
qualification  practices  in  the  simulator.  There  was  no  significant  difference  between  the 
two  groups  based  on  their  qualification  scores  but  the  group  that  conducted  simulator 
training  achieved  a  slightly  higher  pass  rate  after  two  qualification  attempts.  The 
findings  suggest  that  the  additional  simulator  training  had  a  small,  positive  effect  on 
subsequent  live-fire  performance. 

Overall,  however,  it  took  trainees  numerous  attempts  to  qualify  and  it  can  be 
concluded  that  the  additional  simulator  training,  in  conjunction  with  the  initial 
marksmanship  training  was  not  particularly  satisfactory.  Reasons  for  this  outcome 
include  a  lack  of  trainee  experience  with  the  M4  weapon  and  iron  sights,  limitations 
with  the  simulator  target  imagery,  and  few  opportunities  for  individual  coaching.  It  is 
recommended  that  these  issues  are  addressed  in  order  to  develop  a  more  effective 
marksmanship  training  methodology  using  a  combination  of  simulation  and  live-fire. 
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The  data  collected  in  this  study  will  provide  a  basis  for  achieving  this  outcome.  In 
conclusion,  this  study  has  clearly  shown  that  the  simulator  is  only  one  of  many  factors 
that  need  to  be  considered  if  effective  training  outcomes  are  to  be  achieved;  failure  to 
do  so  means  that  questions  regarding  the  appropriate  mix  of  simulator  and  live-fire 
training  cannot  be  answered. 
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1.  Introduction 


Marksmanship  is  a  fundamental  skill  required  by  all  soldiers  which  involves  the  use  of  a 
personal  weapon  to  provide  accurate  and  effective  fire  to  the  maximum  effective  range  of 
that  weapon  [1],  Consequently,  marksmanship  is  a  core  component  of  individual  training, 
both  within  the  Australian  Army  and  within  other  armies  worldwide.  Marksmanship 
training  is  traditionally  conducted  on  live-fire  ranges.  In  particular,  live-fire  qualification  is 
usually  required  to  demonstrate  marksmanship  proficiency  and  achieve  mandated 
training  competencies.  However,  the  cost  of  live  ammunition,  the  availability  and 
accessibility  of  live-fire  ranges,  and  administrative  overheads  associated  with  planning 
and  conducting  live-fire  practices  make  the  use  of  marksmanship  simulators  an  attractive 
alternative  for  marksmanship  training  [2-4],  The  Australian  Defence  Force  (ADF)  has  used 
simulation  for  marksmanship  training  since  2000  [5]1.  Similar  systems  are  currently 
employed  by  the  Canadian  Defence  Force,  US  Army  and  US  Marine  Corps. 

Small  arms  marksmanship  simulators  are  typically  indoor  laser-based  devices  which  can 
support  training  for  a  range  of  weapon  systems,  including  pistols,  rifles,  machine  guns 
and  anti-armour  weapons.  In  these  simulators,  firers  aim  a  modified  weapon  at  a  target 
image  on  a  screen.  When  the  firer  pulls  the  trigger,  the  weapon  fires  a  laser  at  the  target,  a 
camera  detects  the  position  where  the  laser  hits  the  on-screen  target,  and  the  system 
calculates  the  fall-of-shot.  The  system  can  provide  this  information  to  the  firer,  as  well  as 
feedback  on  factors  such  as  the  degree  of  movement  of  the  weapon  barrel,  which  helps 
firers  to  correct  and  refine  their  marksmanship  technique. 

The  use  of  marksmanship  simulators  can  potentially  lead  to  more  efficient  and  effective 
training,  compared  with  traditional  instruction,  through  a  reduction  in  live  training  time 
and  ammunition  usage,  as  well  as  a  measurable  improvement  in  subsequent  live-fire 
performance.  In  addition,  firer  performance  in  the  simulator  can  potentially  be  used  to 
determine  readiness  for  live-fire  qualification.  This  outcome  would  also  lead  to  more 
efficient  training,  as  it  would  help  to  avoid  wasting  ammunition  on  trainees  who  were  not 
ready  to  attempt  live-fire  qualification.  However,  there  is  a  lack  of  empirical  evidence 
regarding  the  efficiency  and  effectiveness  of  simulation-based  marksmanship  training  [6]. 
Consequently,  there  is  a  need  for  further  research  to  address  the  lack  of  data  and  guide  the 
effective  use  of  marksmanship  simulation  in  military  training  programs. 

This  report  documents  the  outcomes  of  a  study  examining  the  use  of  simulation  for  basic 
marksmanship  training  on  an  Australian  Army  training  course.  This  work  was  undertaken 
by  Defence  Science  and  Technology  Organisation  (DSTO)  Land  Operations  Division 
(LOD)  as  part  of  broader  research  program  to  assist  Army  achieve  more  efficient  and 
effective  training  outcomes  through  the  use  of  simulation.  In  support  of  this  requirement, 
the  current  study  investigated  the  utility  of  additional  simulator  training  on  subsequent 
live-fire  qualification  for  a  basic  marksmanship  task  using  the  M4  weapon  with  iron  sights. 
In  this  report,  we  present  empirical  evidence  for  transfer  of  training  from  the  simulator  to 


1  The  ADF  marksmanship  simulator  is  the  Weapons  Training  Simulation  System  (WTSS)  which  is 
manufactured  by  Meggitt  Training  Systems  Inc. 
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the  live-fire  environment.  We  also  demonstrate  that  the  simulator  is  only  one  of  several 
factors  that  contribute  to  effective  training. 


1.1  Previous  research  on  training  efficiency  and  effectiveness  of 
marksmanship  simulators 

Anecdotally,  marksmanship  simulators  are  believed  to  be  effective  environments  for 
trainees  to  become  familiar  with  different  weapon  systems,  as  well  as  learning  the 
principles  of  marksmanship  and  receiving  coaching.  This  is  because  marksmanship 
simulators  are  generally  more  available  and  accessible  than  live-fire  ranges,  as  well  as 
providing  a  benign  environment  in  which  to  learn,  particularly  for  trainees  with  limited  or 
no  experience  with  live-fire  weapons.  The  latter  aspect  is  believed  to  lead  to  faster  skill 
acquisition  and  hence  more  efficient  training  during  the  early  stages  of  learning.  Overall, 
marksmanship  simulation  appears  to  be  viewed  as  an  effective  supplement  (but  not 
replacement)  for  live-fire,  when  used  in  conjunction  with  marksmanship  coaching2. 

Beyond  this  anecdotal  evidence,  there  are  few  studies  that  provide  empirical  data  on  the 
efficiency  and  effectiveness  of  simulation-based  marksmanship  training.  Several  studies 
have  shown  that  firer  performance  is  generally  poorer  in  the  simulator  compared  to  live- 
fire  [4,  7-9].  In  addition,  there  is  considerable  variation  in  the  correlation  between 
simulator  and  live-fire  scores,  with  values  of  the  correlation  coefficient  (r)  ranging  from 
0.17  (low)  to  0.68  (high)  [3,  8-11].  For  the  majority  of  these  studies,  low  to  moderate  values 
of  r  were  found.  In  these  cases,  simulator  performance  was  not  a  useful  predictor  of  live- 
fire  performance  and  it  was  not  possible  to  reliably  predict  live-fire  qualification.  There 
appears  to  be  only  one  example  (r  =  0.68)  where  it  was  possible  to  determine  a  cut-off 
score  in  the  simulator  which  could  be  used  to  predict  live-fire  qualification  with 
reasonable  levels  of  confidence  [9] . 

The  above  findings  have  been  explained  in  part  by  DSTO  research,  which  has  found 
limitations  in  the  simulator  component  technologies  and  system  design  [7,  12],  These 
include  degraded  target  imagery  (relative  to  targets  on  outdoor  firing  ranges)  and 
limitations  in  the  accuracy  of  the  weapon  aim-point  determination.  While  unlikely  to 
impact  on  the  use  of  the  system  for  weapon  familiarisation  and  coaching,  these  factors 
may  limit  the  utility  of  small  arms  simulators  in  terms  of  improving  live-fire  performance, 
particularly  beyond  basic  levels  of  proficiency  [12].  This  issue  is  explored  further  in  the 
following  paragraphs,  which  summarise  the  outcomes  of  six  studies  that  have  attempted 
to  measure  the  training  effectiveness  of  marksmanship  simulators3. 


2  This  view  is  based  on  numerous  discussions  between  the  authors  and  military  subject  matter 
experts  with  considerable  experience  with  both  live-fire  and  simulator  marksmanship  training.  It  is 
also  consistent  with  Army  doctrine  [13]. 

3  In  four  of  these  studies,  the  simulator  was  manufactured  by  the  same  company  (Meggitt  Training 
Systems  Inc.)  and  is  essentially  the  same  device  with  a  different  name.  In  the  fifth  study,  the 
simulator  was  manufactured  by  CUBIC  Training  Systems;  however,  the  system  design  and 
component  technologies  of  the  device  are  essentially  the  same  as  the  other  simulators. 
Consequently,  it  is  appropriate  to  compare  the  outcomes  of  these  studies,  as  they  are  not 
confounded  by  any  major  differences  in  the  simulators  used. 
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In  a  2001  study  with  infantry  trainees,  the  Canadian  Army  assessed  the  effectiveness  of  a 
simulator  (Small  Arms  Trainer  (SAT))  for  training  basic  marksmanship  skills  with  the 
C7A1  rifle  [14],  The  study  compared  the  outcomes  for  the  Level  3  Personal  Weapons  Test4 
(PWT)  of  four  groups  of  soldiers  that  received  basic  marksmanship  training  using  either 
(1)  live-fire  training  only,  (2)  simulator  training  only,  (3)  simulator  training  only  (with  all 
training  repeated),  or  (4)  a  combination  of  simulator  and  live-fire  training.  Each  group 
contained  approximately  30  soldiers.  The  training  involved  conducting  the  'Shoot  to  Live' 
training  program,  which  consists  of  ten  serials,  starting  with  basic  grouping  practices  at 
20  m,  progressing  to  more  advanced  practices  at  longer  ranges  (100  m  -  300  m)  and 
culminating  in  night  shooting  with  illumination  (25  m  -  100  m). 

There  was  no  significant  difference  between  scores  on  the  PWT  for  the  first  three  groups. 
However,  the  fourth  group,  which  received  the  combination  of  simulator  and  live-fire 
training,  achieved  significantly  higher  scores  and  pass  rates  for  the  PWT  than  the  other 
groups.  It  was  concluded  that  the  blended  training  method  allowed  novice  trainees  (with 
minimal  live-fire  experience)  to  focus  on  learning  the  basic  firing  techniques  and  gaining 
familiarity  with  the  weapon  system  on  the  simulator  whilst  conducting  simple 
marksmanship  practices.  Another  significant  finding  was  the  impact  of  marksmanship 
coaching  on  the  live-fire  outcomes.  The  group  which  conducted  training  on  the  SAT  twice 
was  split  into  two  sub-groups,  with  one  group  receiving  minimal  guidance  from  an 
instructor  and  the  other  group  being  actively  coached  and  mentored.  The  group  which 
received  coaching  achieved  higher  scores  on  the  PWT,  consistent  with  the  simulator 
having  limited  effectiveness  as  a  'black  box',  which  automatically  imparts  skill  through 
practice  (i.e.  without  the  need  for  instructor  intervention  or  feedback). 

In  a  2001  study  with  Australian  Army  Reservists,  James  assessed  the  effectiveness  of 
training  aids  in  the  Weapons  Training  Simulation  System  (WTSS)  [l]5.  As  a  secondary 
objective,  the  study  also  measured  the  transfer  of  training  from  the  WTSS  to  the  live-fire 
range  for  the  F88  rifle.  Two  groups  of  soldiers  conducted  a  simple  100  m  grouping  practice 
on  the  live-fire  range  to  establish  baseline  live-fire  marksmanship  ability.  Following  this, 
both  groups  conducted  the  same  grouping  practice  in  the  simulator  twice;  however,  the 
experimental  group  (N=9)  was  provided  additional  feedback  from  the  simulator  training 
aids,  whereas  the  control  group  (N=10)  was  not.  Finally,  both  groups  conducted  the  same 
100  m  practice  on  the  live-fire  range  to  assess  the  degree  of  training  transfer  from  the 
simulator  to  live-fire. 

The  performance  of  the  experimental  group  improved  from  first  to  second  attempt  in  the 
simulator,  whereas  there  was  no  significant  change  in  the  performance  of  the  control 
group,  consistent  with  simulator  training  aids  providing  useful  feedback  to  the  firers. 
Interestingly,  there  was  a  statistically  significant  decrease  in  live-fire  performance  from 
pre-training  to  post-training  for  the  experimental  group,  but  a  significant  increase  in 
performance  for  the  control  group.  Furthermore,  the  effect  sizes  (based  on  Cohens  d)  were 


4  The  Personal  Weapons  Test  Level  3  requires  the  firer  to  engage  the  target  whilst  advancing  at 
ranges  from  400  down  to  25  metres  [15]. 

5  The  WTSS  training  aids  provide  feedback  to  the  firer  such  as  the  fall  of  shot  location  and  the 
degree  of  movement  of  the  weapon  barrel,  which  helps  firers  to  correct  and  refine  their 
marksmanship  technique. 
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medium  in  these  cases  (d  =  0.65  and  d  =  0.54  respectively).  Across  all  firers,  there  was  no 
significant  difference  from  pre-  to  post-training  live-fire  performance.  At  face-value,  these 
findings  suggest  that  there  was  a  negative  transfer  of  training  for  the  experimental  group 
and  a  positive  transfer  of  training  for  the  control  group.  The  author  noted  that  this 
negative  transfer  may  have  occurred  because  the  feedback  provided  by  the  training  aids 
was  not  present  when  the  firers  moved  to  the  live-fire  range,  which  in  turn  may  have 
impacted  negatively  on  their  performance.  While  this  explanation  is  plausible,  the  findings 
from  this  study  should  be  interpreted  with  caution,  given  the  small  sample  sizes  and 
limited  amount  of  practice  trainees  were  given  in  the  simulator. 

In  a  2004  study  with  US  Marine  Corps  recruits,  Yates  examined  the  effectiveness  of  the 
Indoor  Simulated  Marksmanship  Trainer  (ISMT)  for  training  marksmanship 
fundamentals,  compared  with  traditional  methods,  for  the  M16A2  rifle  [16].  An 
experimental  group  (N=55)  conducted  three  days  of  classroom  training  and  two  days  of 
training  in  the  simulator,  followed  by  a  week  of  live-fire  training.  No  specific  details  of  the 
marksmanship  practices  used  during  simulator  training  were  provided.  A  control  group 
(N=55)  conducted  three  days  of  classroom  training  and  two  days  of  dry-fire  training6, 
followed  by  a  week  of  live-fire  training.  After  training,  both  groups  conducted  a  live-fire 
qualification  shoot,  consisting  of  deliberate  and  rapid  fire  at  200,  300  and  500  yards. 

There  was  no  significant  difference  between  the  qualification  scores  of  the  two  groups  and 
it  was  concluded  that  training  on  the  simulator  was  equally  effective  as  dry-fire  training. 
However,  the  author  noted  that  adverse  weather  may  have  degraded  the  performance  of 
the  experimental  group  (which  received  simulator  training)  during  the  qualification  test. 
In  addition,  both  groups  conducted  one  week  of  live-fire  training  after  the  simulator/ dry- 
fire  training,  prior  to  qualification.  This  may  have  had  the  effect  of  reducing  any  difference 
between  the  two  groups  resulting  from  differences  between  simulator  training  and  dry¬ 
firing.  Consequently,  it  would  have  been  preferable  to  compare  both  groups  on  live-fire 
performance  immediately  following  simulator/ dry-fire  training. 

During  a  2009  Australian  Army  training  course,  training  staff  examined  the  effect  of 
additional  simulator  training  on  live-fire  performance  with  the  M4  rifle  [17],  An 
experimental  group  conducted  the  LF6  Application  of  Fire  practice  in  the  simulator  for 
three  hours  prior  to  attempting  the  same  qualification  task  (LF6)  on  the  live-fire  range.  The 
control  group  proceeded  straight  to  live-fire  LF6  qualification.  The  specific  details  of  other 
marksmanship  training  undertaken  by  the  two  groups  are  not  known,  as  this  information 
was  not  reported.  The  study  reported  that  the  experimental  group  achieved  a  100%  pass 
rate  at  their  first  or  second  qualification  attempt,  whereas  the  control  group  achieved  a 
40%  pass  rate,  consistent  with  the  additional  simulator  training  having  a  positive  impact 
on  subsequent  live-fire  performance.  The  training  course  in  the  2009  study  was  the  same 
as  the  course  in  the  current  study.  In  addition,  the  two  studies  used  a  similar  experimental 
design,  which  enables  a  valid  comparison  of  the  outcomes  from  the  two  studies. 


6  Dry  firing  is  a  training  method  involving  firing  without  live  rounds  and  hence  there  is  no  weapon 
recoil  or  noise  from  weapon  discharge.  This  method  allows  the  firer  to  practice  holding  the  weapon 
correctly,  take  a  sight  picture  of  the  target,  and  practice  trigger  release  and  follow-through  without 
using  live  rounds.  The  process  is  observed  by  trained  marksmanship  coaches,  who  provide 
feedback  to  the  trainees  on  their  technique. 
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In  a  2010  study  with  US  Army  cadets,  Merlo,  Frechette,  and  Banks  examined  the  transfer 
of  training  from  the  Engagement  Skills  Trainer7  (EST)  to  live-fire  for  the  M4  rifle  [6].  The 
experimental  group  (N=952)  conducted  marksmanship  training  on  the  EST  prior  to  being 
assessed  on  their  live-fire  performance.  The  control  group  (N=80)  consisted  of  cadets  that 
did  not  train  on  the  EST  but  went  straight  to  live-fire.  During  EST  training,  cadets  fired  at  a 
standard  M4  zeroing  target  at  25  m.  Training  time  in  the  simulator  varied  between 
individuals,  but  did  not  exceed  30  minutes.  During  live-fire  assessment,  cadets  were 
required  to  fire  five  rounds  at  a  target  consisting  of  three  concentric  circles.  Firer 
performance  was  based  on  the  number  of  hits  within  each  circle.  While  the  cadets  fired  up 
to  five  iterations  at  the  target,  only  data  from  the  first  two  iterations  was  analysed,  as  data 
from  subsequent  iterations  was  found  to  be  influenced  by  the  effect  of  the  previous 
iterations.  There  was  no  significant  difference  between  the  performance  of  the  control  and 
experimental  groups;  the  amount  of  time  spent  in  the  simulator  was  also  found  to  have  no 
impact  on  subsequent  live-fire  performance.  These  findings  show  that  the  simulator 
training  had  no  measurable  benefit  in  terms  of  live-fire  performance;  however,  this  finding 
may  be  partly  explained  by  the  limited  amount  of  time  spent  training  in  the  simulator. 

In  a  2012  study  with  US  Navy  personnel,  Jensen  and  Woodsen  [18]  measured  the  training 
effectiveness  of  the  ISMT  for  the  9mm  Berretta  pistol.  In  the  study,  participants  were 
allocated  to  a  control  group  (N=17)  and  an  experimental  group  (N=17)  based  on  previous 
marksmanship  experience.  Both  groups  conducted  a  baseline  performance  assessment  in 
the  simulator;  no  significant  difference  between  the  groups  was  found.  The  control  group 
then  conducted  training  in  the  simulator,  but  with  the  weapon  recoil  disabled,  and 
without  any  fall-of-shot  feedback  (i.e.  dry  firing).  The  experimental  group  conducted  the 
same  training,  but  with  weapon  recoil  enabled,  and  with  fall-of-shot  information  available 
to  trainees.  Training  for  both  groups  utilised  the  same  marksmanship  practices  which 
involved  engaging  targets  at  different  ranges  (3  yards,  7  yards  and  15  yards),  with  no 
timing  restrictions. 

After  training,  both  groups  conducted  a  post-training  assessment  in  the  simulator;  in  each 
case  there  was  a  significant  improvement  from  baseline  scores8.  In  addition,  the 
experimental  group  outperformed  the  control  group,  consistent  with  recoil  and  fall-of-shot 
feedback  resulting  in  better  training  outcomes  within  the  simulator  (similar  to  the  findings 
of  the  James  2001  study  [1]).  Both  groups  then  conducted  a  live-fire  assessment  to  measure 
the  degree  of  transfer  from  the  simulator  to  live-fire.  Measures  of  performance  used  in  the 
study  were  total  scores  (based  on  where  rounds  hit  the  target)  and  mean  point  of  impact 
(MPI)  based  on  the  distance  from  the  true  target  centre  and  the  centre  of  a  group  of  shots 
fired  at  the  target.  There  was  no  significant  difference  between  the  two  groups  on  live-fire 
performance  based  on  total  scores.  However,  there  was  a  statistically  significant  difference 
between  the  groups  based  on  MPI  based  on  one-tailed  significance  testing,  with  medium 
effect  size  (d  =  0.74).  This  outcome  is  similar  to  that  found  by  Yates  [16];  that  is,  the 
simulator  appears  to  provide  similar  (or  slightly  greater)  training  benefit  to  dry-firing. 
However,  Jensen's  study  was  based  on  a  9  mm  pistol,  whereas  the  other  studies  were 


7  The  EST  is  manufactured  by  CUBIC  Training  Systems. 

8  Both  baseline  and  post-training  assessments  required  firers  to  engage  targets  at  various  ranges  (3 
yards,  7  yards  and  15  yards),  under  timed  conditions  and  from  different  firing  positions. 
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based  on  rifles  and  involved  much  longer  target  distances;  it  is  not  clear  how  the  outcomes 
would  generalise  from  one  weapon  system  to  another. 

Overall,  these  studies  provide  some  evidence  that  marksmanship  skills  learned  in  the 
simulator  transfer  to  the  live-fire  environment.  However,  there  are  no  marksmanship 
studies  which  have  examined  the  optimum  blend  of  live-fire  and  simulator  training. 
Consequently,  the  outcomes  of  these  studies  do  not  add  significantly  to  the  anecdotal 
views  regarding  the  training  effectiveness  of  marksmanship  simulators,  or  provide  any 
guidance  on  the  appropriate  mix  of  simulator  and  live-fire  training. 


1.2  The  current  study 

The  current  study  was  conducted  as  part  of  a  broader  research  program  in  DSTO  which 
aims  to  help  Army  achieve  more  efficient  and  effective  use  of  simulation  for  training.  The 
study  was  conducted  during  an  Australian  Army  training  course  in  2010.  During  the 
course,  trainees  undertook  basic  marksmanship  training  on  the  M4  weapon  system  using 
iron  sights9. 

In  order  to  progress  through  the  marksmanship  component  of  the  course,  trainees  were 
required  to  pass  the  LF6  Application  of  Fire  practice  on  the  live-fire  range.  This  practice 
consists  of  eight  serials  and  involves  firing  at  targets  over  different  ranges  (100,  200, 
300  m),  as  well  as  firing  from  different  positions  at  static  and  moving  targets  under  timed 
conditions.  The  practice  is  described  in  more  detail  in  Appendix  A,  together  with  the  other 
basic  marksmanship  practices  used  on  the  course.  The  course  instructors  were  seeking  to 
develop  a  training  program  using  both  simulator  and  live-fire  which  resulted  in  faster  LF6 
qualification  and  higher  marksmanship  standards  within  time  and  resource  constraints 
[19].  This  requirement  was  one  of  the  drivers  for  the  current  study,  but  also  provided  an 
opportunity  to  address  the  lack  of  training  evaluation  studies  in  this  area. 

1.2.1  Training  evaluation  framework 

One  of  the  most  commonly  used  approaches  for  evaluating  training  is  the  Kirkpatrick 
model  [20].  This  model  proposes  training  outcomes  at  four  levels:  reaction  (Level  1), 
learning  (Level  2),  behaviour  (Level  3)  and  results  (Level  4),  as  described  below.  In  the 
current  study,  we  evaluated  training  outcomes  at  all  four  levels. 

•  Level  1:  outcomes  include  assessment  of  participants'  reaction  to  the  training 
program.  In  the  current  study,  we  examined  trainee  feedback  on  what  they 
liked/  disliked  about  simulator  and  live-fire  training. 

•  Level  2:  outcomes  are  quantifiable  indicators  of  learning  that  occur  during  training. 
In  the  current  study,  we  examined  changes  in  trainees'  marksmanship  scores  over 
the  course  of  training. 

•  Level  3:  outcomes  examine  the  extent  to  which  knowledge  and  skills  gained  during 
training  are  applied  (or  transfer)  to  the  job  context;  this  is  also  known  as  transfer  of 


9  This  is  significant  because  anecdotally,  trainees  on  the  course  have  little  prior  experience  with  iron 
sights,  which  has  implications  for  their  marksmanship  performance. 
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training.  In  the  current  study  we  measured  pass  rates  and  changes  in 
marksmanship  competency  levels. 

•  Level  4:  outcomes  provide  a  measure  of  the  impact  that  the  training  has  had  on 
organisational  goals  or  performance.  In  the  current  study  we  measured  the  total 
training  time,  the  amount  of  ammunition  used  and  the  trainee  throughput. 

1.2.2  Study  aims 

The  primary  aim  of  the  study  was  to  measure  the  effect  of  additional  LF6  simulator 
training  on  live-fire  qualification.  The  hypotheses  were  as  follows: 

HO:  There  will  be  no  differences  in  LF6  live-fire  qualification  between  groups  that 
receive  LF6  simulator  training  or  no  simulator  training  prior  to  live-fire 
qualification. 

HI:  The  group  that  receives  LF6  simulator  training  will  have  higher  LF6  live-fire 
qualification  rates  and  scores  compared  with  a  group  that  does  not  receive  LF6 
simulator  training. 

The  second  aim  of  the  study  was  to  investigate  predictors  of  live-fire  LF6  qualification. 
Specifically,  we  examined  the  utility  of  LF6  simulator  scores  and  trainee  demographic  data 
as  predictors  of  live-fire  LF6  qualification. 

The  third  aim  of  the  study  was  to  compare  the  outcomes  with  those  of  the  2009  study  [17]. 
Specifically,  we  compared  the  pass  rates  between  the  two  studies  after  two  LF6 
qualification  attempts. 


2.  Method 


2.1  Study  approval 

The  study  methods  and  procedures  were  approved  by  an  internal  human  research  ethics 
review  committee  prior  to  data  collection  (Ethics  Protocol  Number:  DSTO  LOD  07/10). 
The  study  procedures  were  conducted  in  accordance  with  ethical  principles  and 
guidelines  for  the  conduct  of  human  research  [21]. 


2.2  Participants 

A  range  of  demographic  data  was  collected  from  the  study  participants  using  a  paper- 
based  questionnaire.  This  data  was  collected  as  standard  practice,  in  order  to  characterise 
the  study  population,  as  well  as  identifying  any  potential  confounds  (e.g.  poor  health).  In 
addition,  the  demographic  data  was  used  to  investigate  predictors  of  live-fire  performance 
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and  qualification;  this  is  described  in  Section  2.5.3.  A  copy  of  the  questionnaire  is  included 
in  Appendix  B. 

Thirty-six  ADF  personnel  voluntarily  took  part  in  the  study.  The  age  of  the  participants 
ranged  from  21  to  36  years  (mean  =  26.3  years,  SD  =  4.1  years).  The  average  length  of  their 
military  service  was  2.6  years  (SD  =  3.1  years),  and  the  majority  of  participants  (81%)  had 
an  Army  background. 

In  terms  of  marksmanship  experience,  eight  participants  (22%)  indicated  that  they  had 
used  the  M4  weapon  prior  to  the  course.  Half  the  participants  (50%)  reported  having 
experience  using  iron-sights,  mainly  with  other  weapon  systems  (e.g.  pistols,  shotguns).  In 
terms  of  marksmanship  ability,  25%  of  participants  rated  themselves  as  "Above  Average", 
58%  rated  themselves  as  "Average",  and  17%  rated  themselves  as  "Below  Average". 

All  participants  reported  having  shot  in  either  the  simulator  or  the  live-fire  range  in  the 
last  12  months,  with  the  majority  (86%)  having  shot  at  least  twice  in  both  environments 
during  this  time.  All  participants  had  fired  LF6  at  least  once  previously,  with  the  majority 
(83%)  having  shot  LF6  at  least  twice  within  the  last  3  months  leading  up  to  the  study. 
Overall,  the  participant's  prior  marksmanship  experience  was  limited  but  they  were 
already  familiar  with  the  qualification  task  prior  to  the  study. 

All  participants  reported  having  good  levels  of  general  health  at  the  time  of  the  trial.  Two- 
thirds  (64%)  of  the  participants  reported  having  20/20  vision.  Four  participants  (11%) 
reported  that  they  wore  corrective  eyewear.  The  remaining  nine  participants  reported 
having  non  20/20  vision,  but  did  not  report  wearing  corrective  eyewear.  As  visual  acuity 
testing  was  not  conducted  during  the  study,  the  standard  of  their  eyesight  was  not  known. 


2.3  Study  design 

The  study  design  was  developed  in  conjunction  with  the  instructional  staff.  The  study 
team  were  advised  that  the  following  constraints  applied: 

•  a  total  of  five  days  is  available  for  marksmanship  training  and  LF6  qualification10 

•  two  days  are  allocated  for  LF6  live-fire  qualification  (to  ensure  all  trainees  have  a 
reasonable  chance  of  passing) 

•  three  days  are  available  for  practical  training 

•  all  trainees  must  receive  the  same  amount  of  training11. 

The  implication  for  designing  the  study  methodology  was  that  any  training  intervention 
conducted  by  an  experimental  group  would  also  have  to  be  conducted  by  the  control 


10  During  the  week  prior  to  marksmanship  training,  trainees  received  5  days  of  training  with  the  M4 
weapon.  This  included  classroom  lessons  and  hands-on  training  and  assessment  in  the  handling, 
stripping,  maintenance,  assembly  and  operation  of  the  M4  weapon  system.  The  training  also 
included  lessons  on  marksmanship. 

11  Course  instructors  indicated  that  this  was  necessary  to  avoid  the  risk  of  trainees  that  did  not 
qualify  at  LF6  seeking  a  redress  of  action,  as  failure  to  qualify  at  LF6  would  lead  to  trainees  failing 
the  course  [22]. 
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group  at  some  stage  during  the  week  allocated  to  marksmanship  training.  Consequently, 
the  length  of  the  training  intervention  was  restricted  to  half  a  day;  any  longer  than  this 
may  have  resulted  in  insufficient  time  for  the  control  group  to  qualify  at  LF6. 

Given  these  constraints,  it  was  agreed  with  the  instructional  staff  that  the  study  would 
examine  the  impact  of  an  additional  three  LF6  practices  in  the  simulator  on  subsequent 
LF6  live-fire  qualification.  Specifically,  the  study  compared  the  outcomes  for  the  first  three 
LF6  qualification  attempts  between  the  group  that  received  three  LF6  practices  in  the 
simulator  and  a  group  which  received  no  additional  simulator  training.  The  reasons  for 
this  are  as  follows. 

•  Firstly,  the  time  taken  to  conduct  three  LF6  simulator  practices  is  close  to  the 
maximum  amount  of  training  able  to  be  conducted  in  half  a  day. 

•  Secondly,  three  practices  allow  the  opportunity  for  feedback  and  correction,  to 
enable  learning  to  occur. 

•  Finally,  three  qualification  attempts  are  considered  a  reliable  basis  on  which  to 
compare  the  outcomes  for  the  two  groups. 

The  study  design  is  illustrated  in  Table  1  below.  The  study  was  only  a  component  of  the 
full  training  program  conducted  by  the  two  groups;  this  is  described  in  Section  2.4  and 
presented  in  Table  2. 


Table  1.  Study  design. 


Group  1 

(experimental 

group) 

Initial  Training 

3  LF6  practices 
(simulator) 

3  LF6  qualification  attempts 

Group  2 
(control  group) 

Initial  Training 

No  additional 
training 

3  LF6  qualification  attempts 

In  addition,  it  was  agreed  with  the  instructional  staff  that  the  study  would  not  include  any 
individual  marksmanship  coaching,  in  order  to  remove  a  potential  confound  (i.e.  ensuring 
equivalent  levels  of  coaching/ feedback  across  the  two  groups).  Overall,  it  was  not 
anticipated  that  this  study  would  yield  an  optimum  training  solution.  However,  it  was 
expected  that  the  study  would  provide  insights  into  the  benefit  provided  by  simulator 
training  and  provide  a  basis  for  developing  an  optimum  training  methodology  as  part  of 
future  studies. 


2.4  Procedure 

On  the  first  day,  the  study  team  briefed  the  training  staff  and  participants  on  the  study  aim 
and  procedures.  Following  the  brief,  the  participants  completed  the  demographic 
questionnaire.  The  questionnaire  requested  details  regarding  participants'  age,  length  of 
service,  prior  marksmanship  experience,  self-rating  of  marksmanship  skill  level,  general 
health  status,  visual  acuity,  and  confidence  using  the  M4  and  passing  LF6. 
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The  study  schedule  is  outlined  in  Table  2.  Prior  to  marksmanship  training,  all  trainees 
were  required  to  zero  their  weapon  on  a  25  m  firing  range.  On  Day  1  and  2,  all  participants 
conducted  initial  training  on  the  live-fire  range  and  in  the  simulator.  The  training  content 
and  delivery  is  at  the  discretion  of  the  training  staff  but  generally  involves  conducting  a 
number  of  basic  marksmanship  practices  as  preparation  for  LF6  qualification.  These 
practices  can  be  conducted  in  both  the  simulator  and  on  the  live-fire  range  and  are 
described  in  more  detail  in  Appendix  A12. 

Following  training,  the  participants  were  split  into  two  groups  on  the  basis  of  their  live- 
fire  scores  for  the  LF1  practice  conducted  on  Day  1.  This  was  achieved  by  entering  the 
participants'  names  and  corresponding  LF1  grouping  scores  into  a  spreadsheet,  ordering 
the  scores  from  lowest  to  highest,  and  then  allocating  participants  to  the  two  groups  on  an 
alternating  basis.  The  resultant  mean  LF1  scores  for  each  group  were  172.2  mm  and 
172.2  mm,  which  were  not  significantly  different  (mean  difference  =  0.0  mm,  t  =  0.00,  p  = 
1.0)  based  on  an  independent  samples  t-test. 

On  Day  3,  Group  1  (experimental  group)  received  the  additional  simulator  training; 
specifically,  they  conducted  three  LF6  practices  in  the  simulator  with  no  coaching.  At  the 
same  time.  Group  2  (control  group)  went  straight  to  the  live-fire  range  and  conducted 
three  attempts  at  LF6  qualification.  For  both  the  simulator  and  live-fire  practices,  there 
were  breaks  provided  to  all  participants  between  LF6  practices  in  order  to  reduce  the 
likelihood  of  fatigue  effects  on  participant's  performance.  In  the  afternoon,  the 
experimental  group  conducted  three  LF6  qualification  attempts  on  the  live-fire  range13. 
While  the  experimental  group  was  conducting  LF6  qualification,  the  control  group 
conducted  three  LF6  practices  in  the  simulator  with  no  coaching.  This  was  not  part  of  the 
study  design  but  was  simply  to  ensure  that  the  control  group  was  not  disadvantaged  and 
received  the  same  amount  of  training  (as  discussed  in  Section  2.3).  While  not  a  specific 
study  aim,  this  experimental  design  allowed  the  effect  of  live-fire  on  subsequent  LF6 
performance  in  the  simulator  (i.e.  reverse  transfer  of  training)  to  be  measured,  by 
comparing  firer  performance  on  the  three  LF6  simulator  practices  for  the  two  groups.  Note 
that  the  LF6  simulator  practices  did  not  count  towards  qualification  as  this  occurs  only  on 
the  live-fire  range.  On  days  4  and  5,  both  groups  continued  to  conduct  LF6  qualification 
until  all  trainees  qualified.  All  trainees  conducted  at  least  four  qualification  attempts14. 

All  LF6  scores  for  the  simulator,  and  for  the  first  three  live-fire  qualification  attempts, 
including  scores  for  each  of  the  eight  serials,  were  recorded  for  all  participants  at  the 


12  For  the  current  study,  training  involved  LF1,  LF4  and  LF5;  LF2  and  LF3  were  not  conducted,  due 
to  time  constraints  and  range  availability. 

13  Due  to  time  limitations,  the  experimental  group  actually  completed  their  third  LF6  practice  on  the 
live-fire  range  the  following  day. 

14  Trainees  only  need  to  qualify  on  LF6  once  on  the  course  to  progress  through  the  marksmanship 
training.  Flowever,  in  this  study  we  assessed  their  level  of  competency  over  three  attempts;  this 
meant  that  some  trainees  qualified  more  than  once.  Instructional  staff  also  indicated  that  it  was 
preferable  for  trainees  to  achieve  a  qualification  score  more  than  once  in  order  to  confirm  their 
competence  at  LF6. 
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completion  of  each  practice15.  The  weather  conditions  were  fine  with  very  light  winds  for 
all  shooting  activities  conducted  on  the  live-fire  range  throughout  the  study.  For  all 
practices,  the  participants  wore  the  Australian  Army  camouflage  pattern  uniform  and 
webbing  as  per  the  course  requirements. 


Table  2.  Training  and  qualification  schedule  conducted  by  Group  1  and  Group  216.  Tire  study 
design  is  enclosed  in  the  bold  border. 


Time 

Group  1:  Experimental  Group 

Group  2:  Control  Group 

pre- training 

25  m  zeroing 

25  m  zeroing 

Day  1 

LF1  (live-fire  range)  4-8  attempts 

LF1  (live-fire  range)  4-8  attempts 

LF1  (simulator)  1  attempt 

LF1  (simulator)  1  attempt 

Day  2 

LF4  (simulator)  1  attempt 

LF4  (simulator)  1  attempt 

LF5  (simulator)  1  attempt 

LF5  (simulator)  1  attempt 

Day  3 

LF6  (simulator)  3  attempts 

LF6  qualification  (live-fire  range)  3 
attempts 

LF6  qualification  (live-fire  range)  3 
attempts 

LF6  (simulator)  3  attempts 

Day  4  and  5 

LF6  qualification  (live-fire  range) 
until  all  trainees  qualify 

LF6  qualification  (live-fire  range) 
until  all  trainees  qualify 

2.5  Measures 

2.5.1  LF6  performance 

The  primary  measure  of  marksmanship  performance  used  in  the  study  was  the  total 
number  of  points  scored  on  the  LF6  practice.  Five  points  are  allocated  for  each  hit  on  the 
target;  the  highest  possible  score  is  235  points  and  a  pass  score  is  165  points  (i.e.  70%  of 
maximum  score);  achieving  165  points  on  the  live-fire  range  results  in  qualification. 

2.5.2  Simulator  training  effectiveness 

A  number  of  different  measures  were  used  to  assess  the  effectiveness  of  the  additional 
simulator  training  (Level  3): 

•  The  difference  in  LF6  live-fire  qualification  rates  achieved  for  each  group  after  each 
qualification  attempt. 

•  The  difference  in  mean  live-fire  LF6  scores  from  the  first  three  qualification 
attempts  on  the  live-fire  range  for  both  groups. 


15  The  study  team  collected  live-fire  serial  scores  for  the  first  three  qualification  attempts  only;  total 
scores  for  the  remaining  LF6  practices  were  collected  and  forwarded  to  the  study  team  by  the 
instructional  staff. 

16  LF1  is  a  grouping  practice  in  which  performance  is  measured  as  the  extreme  spread  between  the 
two  shots  furthest  apart  in  a  group  of  shots.  For  this  practice,  lower  scores  indicate  better 
marksmanship  performance.  LF4,  LF5  and  LF6  are  application  of  fire  practices,  where  higher  scores 
indicate  better  performance.  These  practices  are  described  in  detail  in  Appendix  A. 
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•  Transfer  effectiveness  ratio  (TER)  [23-24],  The  TER  is  a  measure  of  the  time  or 
number  of  repetitions  in  live  training  (required  to  reach  competency)  saved  by  the 
use  of  simulator  training,  relative  to  the  length  of  time  or  number  of  repetitions  of 
simulator  training  required  to  achieve  this  saving.  In  this  study,  the  experimental 
group  conducted  three  repetitions  of  LF6  in  the  simulator,  hence: 

N1-N2 

TER  = -  Equation  1 

N  3  ' 

where: 

o  N1  is  the  mean  number  of  attempts  required  for  the  control  group  (i.e. 
Group  2)  to  pass  LF6  (no  simulation) 

o  N2  is  the  mean  number  of  attempts  required  for  the  experimental  group 
(i.e.  Group  1)  to  pass  LF6 

o  N3  is  the  number  of  attempts  at  LF6  (3)  in  the  simulator  for  the 
experimental  group. 

•  Percent  transfer  (PT)  [23-24],  The  PT  is  a  measure  of  the  time  or  number  of 
repetitions  in  live  training  (required  to  reach  competency)  saved  by  the  use  of 
simulation  based  training,  relative  to  the  length  of  time  or  number  of  repetitions  of 
live  training  normally  required  to  achieve  competency  (expressed  as  a  percentage). 

N1—N2 

PT  =  100  x -  Equation  2 

N 1  ' 

The  effect  of  live-fire  on  subsequent  LF6  performance  in  the  simulator  was  determined 
using  similar  measures  (i.e.  comparing  mean  LF6  scores  and  pass  rates  in  the  simulator  for 
both  groups).  However,  as  qualification  occurs  only  on  the  live-range,  TER  and  PT  are  not 
relevant  measures  in  this  case. 

2.5.3  Predictors  of  LF6  qualification 

Linear  regression  was  used  to  investigate  predictors  of  live-fire  qualification.  Simple  linear 
regression  was  used  to  investigate  the  utility  of  simulator  LF6  scores  in  predicting  LF6 
qualification  scores.  Multiple  linear  regression  was  used  to  investigate  the  utility  of 
additional  predictors  of  live-fire  LF6  scores.  The  following  variables  (collected  from  the 
demographic  questionnaire)  were  included  in  the  regression  model.  Categorical  variables 
(e.g.  prior  experience  with  iron  sights)  which  could  take  only  two  possible  values  were 
excluded. 

1.  simulator  scores  (1st  attempt) 

2.  simulator  scores  (2nd  attempt) 

3.  simulator  scores  (3rd  attempt) 

4.  mean  simulator  scores  (over  3  attempts) 

5.  number  of  times  fired  in  simulator  (past  12  months) 

6.  number  of  times  fired  live  (past  12  months) 

7.  marksmanship  skill  level  (1  =  below  average,  2  =  average,  3  =  above  average) 

8.  number  of  times  conducted  LF6  (prior  to  study) 

9.  time  since  last  fired  LF6  (months) 

10.  amount  of  experience  with  M4  (months) 
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11.  confidence  in  using  M4  (low  =  1,  moderate  =  2,  high  =  3) 

12.  confidence  in  passing  LF6  first  time  (low  =  1,  moderate  =  2,  high  =  3) 

13.  general  health  during  study  (1  =  well  below  average,  2  =  below  average,  3  = 
average,  4  =  good,  5  =  excellent). 

2.5.4  Number  of  rounds  in  training  and  qualification 

The  following  measures  were  used  as  Level  4  outcomes  under  the  Kirkpatrick  framework 
as  they  relate  to  the  cost  and  efficiency  of  training: 

•  total  number  of  live-rounds  expended  in  training  and  qualification 

•  proportion  of  simulator  rounds  fired  in  training 

•  total  number  of  training  rounds  fired  (simulator  plus  live) 

•  average  and  total  number  of  live-fire  qualification  attempts. 

2.5.5  Trainee  reactions 

At  the  conclusion  of  the  study,  the  participants  were  asked  the  following  questions 
regarding  the  training: 

1.  How  did  you  find  shooting  in  the  simulator  as  preparation  for  LF6  on  the  live-fire 
range? 

2.  Did  you  have  any  particular  issues  with  specific  LF6  serials  in  the  simulator  or  live- 
fire  range? 

3.  How  did  you  find  shooting  with  the  M4  weapon  using  iron  sights? 

The  participants'  responses  to  these  questions  are  given  in  Appendix  G  and  Section  3.4.2. 


2.6  Data  analysis 

All  data  analysis  was  conducted  using  SPSS  Version  17.0  [25].  The  data  was  checked  for 
any  missing  or  extreme  values  that  may  have  unduly  affected  the  outcomes  of  the  data 
analysis.  Normality  testing  was  conducted  on  the  data  using  the  Shapiro-Wilk  test.  Mean 
scores  and  standard  deviations  were  calculated  for  all  LF6  serials  and  for  total  LF6  scores. 
Qualification  rates  were  calculated  using  165  points  as  the  cut-off  score.  The  TER  and  PT 
were  calculated  using  Equations  1  and  2. 

Significance  levels  for  differences  in  qualification  rates  between  the  two  groups  were 
determined  using  a  Chi-squared  test  based  on  the  qualification  frequency,  with  an 
acceptance  (p)  value  of  0.05;  the  corresponding  effect  size  was  measured  using  the  Phi 
coefficient.  Where  data  was  found  to  be  normally  distributed,  the  mean  LF6  scores  were 
compared  for  the  two  groups  using  independent  samples  t-tests,  with  an  acceptance  (p) 
value  of  0.05;  the  corresponding  effect  size  was  measured  using  Cohens  d.  Paired  samples 
t-tests  were  used  for  within  group  comparisons  of  mean  LF6  scores.  In  some  cases,  data 
was  non-normally  distributed;  in  this  case,  non-parametric  tests  were  employed  (Mann- 
Whitney  test  for  independent  samples;  Wilcoxon  Signed-rank  test  for  paired  samples). 
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Multiple  linear  regression  was  conducted  using  the  'Forward'  entry  method.  An 
acceptance  (p)  value  of  0.05  was  used  for  both  simple  and  multiple  linear  regression. 


2.7  Feedback 

While  there  was  no  specific  coaching  during  the  study,  some  feedback  was  provided  to 
trainees  while  conducting  LF6  practices  in  the  simulator  and  on  the  live-fire  range.  When 
conducting  LF6  on  the  live-fire  range,  at  the  start  of  Serial  1,  each  trainee  is  allowed  two 
sighting  rounds  to  see  their  fall-of-shot  on  the  target  and  adjust  their  point-of-aim  if 
necessary.  In  addition,  targets  fall  when  hit  for  all  snap  and  moving  target  serials,  which 
provides  trainees  with  performance  feedback.  The  only  form  of  coaching  provided  on  the 
live-fire  range  was  notifying  trainees  when  they  had  not  adopted  the  correct  firing 
position  for  a  particular  serial.  When  conducting  LF6  in  the  simulator,  trainees  were  able 
to  see  their  fall-of-shot  after  each  serial,  which  is  the  default  condition  in  the  simulator. 
However,  the  trainees  were  not  provided  with  any  corrective  action  from  training  staff 
based  on  this  information.  The  training  staff  merely  prompted  participants  to  apply  the 
marksmanship  principles  during  each  practice  and  to  adjust  their  point-of-aim  at  the 
target  if  required. 


3.  Results 


In  this  section,  results  are  presented  in  the  order  to  which  they  relate  to  the  study  aims.  All 
references  to  mean  scores  in  this  section  refer  to  the  average  of  three  attempts  at  LF6 
qualification  unless  stated  otherwise.  Results  based  on  the  first  three  qualification  attempts 
are  presented  first,  followed  by  results  based  on  all  qualification  data,  in  accordance  with 
the  study  design. 


3.1  Effect  of  additional  simulator  training  on  live-fire  qualification 

3.1.1  Qualification  rates  and  mean  scores 

The  qualification  rates  for  LF6  for  the  first  three  attempts  are  shown  in  Table  3.  After  three 
attempts,  58.3%  of  all  trainees  had  passed.  The  group  that  conducted  additional  training  in 
the  simulator  (experimental  group)  passed  more  quickly  than  the  group  that  went  straight 
to  the  live-fire  range  (control  group).  The  difference  in  qualification  rates  between  the  two 
groups  was  statistically  significant  after  the  second  attempt  (y2  =  4.43,  p  =  0.04  (df  =  1), 
<|)  =  0.35).  On  the  basis  of  pass  rates,  it  can  be  concluded  that  the  additional  simulator 
training  provided  a  small  positive  training  benefit.  Inspection  of  Table  3  shows  that  pass 
rates  for  the  experimental  group  after  the  first  qualification  attempt  (i.e.  after  three 
simulator  and  one  live-fire  LF6)  are  the  same  as  the  pass  rates  for  the  control  group  after 
two  qualification  attempts.  Pass  rates  for  the  experimental  group  after  the  second 
qualification  attempt  (i.e.  after  three  simulator  and  two  live-fire  LF6)  are  the  same  as  the 
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pass  rates  for  the  control  group  after  three  qualification  attempts.  On  this  basis,  it  can  be 
speculated  that  the  three  practices  in  the  simulator  provided  (roughly)  the  equivalent 
training  benefit  as  a  single  qualification  attempt  on  the  live-fire  range  for  this  sample  of 
trainees. 


Table  3.  Qualification  rates  for  LF6  on  the  live-fire  range  for  three  attempts. 


Group 

Qualification  rate  (%) 

Number  of 

passes  achieved  in  three 
attempts  (%) 

First 

Second 

Third 

Passed 

Passed 

Passed 

attempt 

attempt 

attempt 

once 

twice 

three  times 

Experimental 

27.8 

50.0 

66.7 

38.9 

5.6 

22.2 

group 

Control  group 

22.2 

27.8 

50.0 

22.2 

11.1 

16.7 

All 

participants 

25.0 

38.9 

58.3 

30.6 

8.3 

19.4 

The  mean  scores  for  all  participants,  as  well  as  those  for  each  group  are  shown  in  Table  4. 
The  difference  between  the  mean  scores  over  three  attempts  for  the  two  groups  is  8.2 
points;  this  was  not  statistically  significant  ( t  =  0.91,  p  =  0.37).  This  indicates  that  the  impact 
(either  positive  or  negative)  of  additional  simulator  training  was  too  small  to  be  reliably 
detected  using  mean  scores  as  the  measure  of  effectiveness. 


Table  4.  Mean  scores  for  LF6  practices.  Numbers  in  brackets  are  one  standard  deviation.  The  pass 
score  is  165  points  and  maximum  possible  score  is  235  points. 


Group 

First 

attempt 

Second 

attempt 

Third 

attempt 

Average  of 
three 
attempts 

Experimental 

147.5 

149.2 

159.7 

152.1 

group 

(29.0) 

(27.1) 

(19.7) 

(21.7) 

Control  group 

140.8 

139.2 

151.7 

143.9 

(34.1) 

(33.1) 

(38.4) 

(31.5) 

All 

144.2 

144.2 

155.7 

148.0 

participants 

(31.3) 

(30.2) 

(30.4) 

(27.0) 

The  mean  scores  for  each  LF6  serial  are  given  in  Table  5  for  all  participants,  as  well  as 
those  for  each  group.  Participants  performed  better  at  the  100  m  serials  (Serials  6,  7  and  8) 
than  those  at  200  m  (Serials  3, 4  and  5)  and  300  m  (Serials  1  and  2).  This  is  consistent  with  it 
being  more  difficult  to  hit  targets  at  longer  ranges.  The  difference  between  the  scores  for 
the  two  groups  approached  statistical  significance  for  Serial  3  (ii  =  101.5,  z  =  -1.924,  p  = 
0.054).  This  outcome  suggests  that  the  additional  simulator  training  was  effective  in  terms 
of  subsequent  live-fire  performance  for  this  serial.  However,  the  difference  is  only  3.7 
points  which  is  less  than  one  hit  on  the  target  and  is  therefore  of  no  practical  significance. 
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Table  5.  Mean  LF6  live-fire  scores  for  each  serial.  Numbers  in  brackets  are  one  standard  deviation. 
HPS  is  30  points  (Serials  1-7)  and  25  points  (Serial  8). 


Group 

Serial 

1 

2 

3 

4 

5 

6 

7 

8 

deliberate, 
300  m 

snap, 
300  m 

rapid, 
200  m 

snap, 

200  m 

rapid, 
200  m 

snap, 
100  m 

snap, 
100  m 

moving 

target 

Experimental 

group 

18.3 

(6.2) 

14.7 

(5.7) 

18.2 

(6.8) 

15.6 

(5.9) 

13.9 

(4.7) 

26.4 

(2.9) 

22.9 

(4.5) 

21.9 

(3.3) 

Control 

16.6 

13.0 

14.5 

15.5 

14.6 

26.9 

21.7 

21.1 

group 

(6.4) 

(6.3) 

(5.7) 

(7.3) 

(6.1) 

(2.9) 

(4.2) 

(4.5) 

All 

17.5 

13.8 

16.4 

15.6 

14.3 

26.7 

22.3 

21.5 

participants 

(6.2) 

(6.0) 

(6.5) 

(6.5) 

(5.4) 

(2.9) 

(4.3) 

(3.9) 

The  pass  rates  for  all  LF6  qualification  attempts  are  shown  in  Figure  1.  Differences  in 
qualification  rates  observed  during  the  first  three  attempts  are  maintained  for  subsequent 
attempts;  these  were  statistically  significant  when  comparing  qualification  rates  after  four 
and  five  attempts  (y2  =  5.63,  p  =  0.02  (df  =  1),  4>  =  0.40  and  y2  =  4.43,  p  =  0.04  (df  =  1),  <|>  = 
0.35,  respectively). 


■  Group  1 
(simulator 
training) 

□  Group  2  (no 
simulator 
training) 


Figure  1.  Pass  rates  for  both  groups  based  on  LF6  qualification  attempts. 


The  control  group  conducted  three  simulator  LF6  practices  between  their  third  and  fourth 
qualification  attempt  on  the  live-fire  range.  The  resulting  increase  of  12.5  points  in  the 
mean  scores  was  not  statistically  significant  ( t  =  1.55,  p  =  0.14).  Furthermore,  it  should  be 
noted  that  this  difference  is  biased  by  the  performance  of  one  participant,  who  improved 
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by  110  points  from  their  third  to  fourth  attempt17;  if  the  scores  for  this  participant  are 
removed,  the  difference  is  reduced  to  6.8  points  (t  =  1.13,  p  =  0.28).  Indeed,  the  three  LF6 
practices  conducted  in  the  simulator  by  the  control  group  resulted  in  only  one  additional 
trainee  passing  LF6  at  the  4th  attempt  on  the  live-fire  range.  Overall,  these  outcomes 
indicate  that  the  simulator  training  conducted  by  the  control  group  (between  their  third 
and  fourth  qualification  attempts)  had  little  impact  on  subsequent  live-fire  performance. 

3.1.2  Transfer  Effectiveness  Ratio  and  Percent  Transfer 

The  final  Level  3  measures  of  effectiveness  for  the  additional  simulator  training  are  TER 
and  PT.  The  mean  number  of  attempts  for  the  experimental  group  to  qualify  (N2)  was  2.8 
(SD  =  1.6);  the  corresponding  value  for  the  control  group  (Nl)  was  4.1  (SD  =  2.6).  Values  of 
TER  (0.43)  and  PT  (31.5%)  were  not  statistically  significant  (U  =  115.0,  z  =  -1.51,  p  =  0.13), 
consistent  with  any  transfer  of  training  from  the  simulator  to  live-fire  being  too  small  to  be 
reliably  detected  based  on  these  measures  of  effectiveness. 

It  should  be  remembered  that  the  control  group  was  required  to  conduct  three  LF6 
practices  in  the  simulator  between  their  third  and  fourth  attempts  at  LF6  on  the  live-fire 
range  (in  order  to  ensure  both  groups  conducted  the  same  amount  of  training).  If  this 
training  was  beneficial,  then  this  would  result  in  an  underestimation  of  Nl  and  hence  an 
underestimation  of  TER  and  PT.  However,  as  noted  above,  there  is  evidence  to  suggest 
that  the  simulator  training  conducted  by  the  control  group  had  little  impact  on  subsequent 
live-fire  performance;  hence  it  is  likely  that  any  underestimation  of  TER  and  PT  is 
minimal. 

3.1.3  Rounds  used  during  training 

The  number  of  rounds  fired  during  training  by  the  two  groups  is  shown  in  Table  6.  The 
experimental  group  fired  67%  more  rounds  than  the  control  group  during  training.  Given 
that  this  additional  training  yielded  only  a  small  benefit  in  terms  of  qualification 
outcomes,  this  does  not  represent  a  good  return  on  investment  for  Army  (i.e.  at  Level  4). 
While  expending  simulator  rounds  does  not  incur  the  same  cost  as  expending  live  rounds, 
there  is  an  associated  cost  in  the  time  taken,  and  use  of  resources  (using  the  facility  and  the 
simulator  operator,  use  of  training  staff). 


Table  6.  Number  of  simulator  and  live  rounds  fired  during  training 


Group 

Simulator 

Live-fire 

Total 

Experimental  group 

5382  (70%) 

2268  (30%) 

7650 

Control  group18- 19 

2196  (48%) 

2388  (52%) 

4584 

17  The  next  highest  increase  was  40  points. 

18  When  the  additional  3  LF6  simulator  practices  conducted  by  the  control  group  are  taken  into 
account,  this  figure  becomes  69%  simulator  training  (5382  simulator  rounds)  and  31%  live-fire 
training  (2388  live  rounds). 

19  The  control  group  conducted  slightly  more  LF1  practices  on  the  live-fire  range  than  the 
experimental  group  (on  average  5.3  versus  5.0  practices  per  trainee);  this  difference  was  not 
statistically  significant  (LI  =  145.5,  z  =  -0.58,  p  =  0.56). 
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3.1.4  Number  of  live-fire  qualification  attempts 

On  average,  trainees  took  3.4  attempts  to  pass  the  LF6  practice.  Overall,  it  took  ten 
attempts  over  two-and-a-half  days  for  all  trainees  to  qualify  (in  addition  to  the  three  LF6 
practices  conducted  in  the  simulator).  Consequently,  the  training  method  used  in  this 
study  (Table  2)  was  not  particularly  effective,  and  improvements  are  required.  Additional 
analyses  were  conducted  in  order  to  identify  potential  training  improvements;  these  are 
presented  in  Appendix  C.  It  was  found  that  firers  who  struggled  to  qualify  had  the  most 
difficulty  with  targets  at  200  m  and  300  m.  In  addition,  it  was  found  that  trainees  found  it 
easier  to  hit  targets  at  longer  ranges  on  the  live-fire  range  than  they  did  in  the  simulator. 
The  implications  of  these  outcomes  are  discussed  later  in  Section  4.5,  together  with  options 
for  improving  training. 


3.2  Predictors  of  LF6  qualification 

The  ability  to  predict  marksmanship  performance  can  be  useful  for  identifying  those 
trainees  who  are  likely  to  qualify  and  those  who  may  require  additional  training.  In  both 
cases,  such  information  can  help  inform  decisions  regarding  training  scheduling  to 
prevent  resource  wastage.  In  this  study,  a  number  of  measures  were  investigated  for  their 
utility  as  predictors  of  LF6  qualification;  the  results  are  presented  in  detail  in  Appendix  D. 
The  correlation  between  mean  live-fire  and  simulator  LF6  scores  (over  three  attempts)  for 
all  participants  was  moderate  in  size  (r  =  0.40),  and  was  statistically  significant  (p  =  0.02)20. 
Conducting  multiple  linear  regression  using  additional  predictor  variables  resulted  in 
larger  correlation  coefficients  (up  to  r  =  0.73),  however  this  did  not  result  in  a  reliable 
prediction  of  LF6  qualification.  In  contrast,  participants'  self-rating  of  marksmanship  skill 
level  showed  some  utility  in  predicting  qualification.  The  majority  of  firers  who  rated 
themselves  as  'Below  Average'  shooters  failed  to  pass  any  of  their  first  four  qualification 
attempts21.  Conversely,  all  firers  who  rated  themselves  'Above  Average'  shooters  passed 
at  least  two  of  their  first  four  qualification  attempts.  These  findings  are  discussed  further 
in  Section  4.2. 


3.3  A  comparison  of  study  outcomes  with  the  2009  course 

As  noted  in  Section  1.1,  the  outcomes  of  a  similar  study  conducted  during  the  2009  course 
indicated  that  a  group  that  conducted  3  hours  training  in  the  simulator  prior  to  conducting 
live-fire  (experimental  group)  achieved  a  100%  qualification  rate  for  LF6  after  the  first  or 
second  attempt;  the  corresponding  rate  for  the  group  that  went  straight  to  the  live-fire 
range  (control  group)  was  40%  [17],  For  the  current  study,  the  experimental  group 
achieved  a  pass  rate  of  50  %  after  two  attempts  at  LF6;  the  control  group  achieved  a  pass 
rate  of  28%  at  the  same  stage.  At  face- value,  these  outcomes  appear  quite  different  from 


20  The  correlation  coefficient  was  calculated  using  the  mean  LF6  scores  over  three  attempts  as  this 
provides  a  more  reliable  measure  of  firer  performance. 

21  Trainees  only  need  to  qualify  on  LF6  once  on  the  course  to  progress  through  the  marksmanship 
training.  However,  in  this  study  we  assessed  their  level  of  competency  over  three  attempts;  this 
meant  that  some  trainees  qualified  more  than  once. 
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those  of  the  2009  study.  However,  an  analysis  of  the  data  shows  that  the  apparent 
differences  can  be  explained  by: 

•  differences  in  the  baseline  marksmanship  ability  levels  between  trainees  in  the  two 
trials  (resulting  from  differences  in  training  methods  between  the  two  courses) 

•  the  use  of  pass  rates  as  a  measure  of  effectiveness 

•  small  sample  sizes. 

These  analyses  are  presented  in  more  detail  in  Appendix  E  and  the  outcomes  discussed  in 
Section  4.3. 


3.4  Additional  data 

The  preceding  sections  presented  the  results  specifically  addressing  the  study  aims,  and 
included  an  evaluation  of  training  corresponding  to  levels  3  and  4  of  the  Kirkpatrick 
framework.  The  following  sections  present  additional  data,  including  an  assessment  of  the 
reverse  transfer  of  training  (from  live-fire  to  simulator),  as  well  as  training  evaluation  at 
Levels  1  and  2  of  the  Kirkpatrick  framework. 

3.4.1  Reverse  transfer  of  training 

The  effect  of  live-fire  LF6  on  subsequent  simulator  performance  was  measured  by 
comparing  pass  rates  (i.e.  scores  >  165  points)  and  mean  LF6  scores  for  the  two  groups, 
based  on  their  performance  in  the  simulator  (see  Section  2.5.2).  These  results  are  presented 
in  detail  in  Appendix  F.  There  was  no  significant  difference  in  pass  rates  and  mean  scores 
in  the  simulator  between  the  two  groups.  This  demonstrates  that  the  three  LF6  practices 
conducted  by  the  control  group  on  the  live-fire  range  had  no  measurable  benefit  on  their 
subsequent  performance  in  the  simulator.  That  is,  there  was  no  detectable  reverse  transfer 
of  training  from  live-fire  to  the  simulator  for  the  LF6  practice.  Reasons  for  this  outcome  are 
discussed  in  Section  4.4. 

3.4.2  Participant  feedback 

Participant  feedback  corresponds  to  Level  1  in  the  Kirkpatrick  training  evaluation 
framework  and  provides  a  qualitative  measure  of  training  effectiveness,  as  well  as 
identifying  areas  for  improvement.  The  comments  made  by  the  participants  (and 
instructional  staff)  in  response  to  the  questions  asked  at  the  conclusion  of  the  study  are 
presented  in  Appendix  G  and  are  summarised  here.  Overall,  participants  indicated  that 
the  simulator  provided  a  useful  environment  for  rehearsing  the  LF6  task  prior  to 
qualification.  However,  a  number  of  issues  were  raised  as  follows: 

•  Participants  reported  having  difficulty  seeing  the  LF6  targets  in  the  simulator, 
particularly  at  longer  ranges  (e.g.  200  m  and  300  m).  This  observation  was 
consistent  with  the  poorer  firer  performance  in  the  simulator  compared  with  the 
live-fire  range  at  these  distances. 

•  Participants  noted  that  the  recoil  force  of  the  simulated  M4  weapon  is  lower  than 
the  recoil  force  of  the  real  M4  weapon,  which  could  potentially  have  an  impact  on 
the  time  required  to  re-establish  the  point-of-aim  after  firing.  Consequently,  this 
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could  lead  to  a  negative  training  effect  when  moving  from  the  simulator  to  the  live- 
fire  range. 

•  Half  of  the  participants  felt  that  conducting  six  LF6  practices  in  one  day  (three  in 
the  simulator  plus  three  qualification  attempts)  was  excessive.  However,  the 
amount  of  firing  involved  was  not  significantly  different  from  that  typically 
conducted  during  the  course.  In  addition,  there  were  significant  breaks  between 
each  LF6;  consequently,  fatigue  is  unlikely  to  have  affected  the  study  outcomes. 

•  Participants  indicated  that  they  would  have  preferred  to  receive  specific  feedback 
about  their  LF6  performance  on  the  live-fire  range.  While  coaching  is  not  permitted 
during  LF6  qualification,  these  comments  are  consistent  with  the  majority  of 
trainees  having  difficulty  qualifying,  and  hence  needing  additional  assistance. 

•  All  participants  agreed  that  acquiring  and  engaging  targets  using  iron  sights  is 
more  difficult  than  using  optical  sights,  consistent  with  the  slow  rate  of  LF6 
qualification. 

3.4.3  Changes  in  marksmanship  performance  during  training 

Changes  in  marksmanship  performance  during  training  correspond  to  Level  2  in  the 
Kirkpatrick  training  evaluation  framework.  In  this  study.  Level  2  evaluation  of  training 
was  conducted  by  examining  changes  in  marksmanship  scores  where  trainees  conducted 
multiple  attempts  of  the  same  practice.  This  occurred  for  the  LF1  practice  (conducted  on 
the  live-fire  range  on  Day  1  of  the  study),  as  well  as  the  LF6  practices  conducted  in  the 
simulator. 

All  trainees  conducted  at  least  four  attempts  at  LF1  on  the  live-fire  range.  For  all 
participants,  there  was  a  decrease  in  mean  group  size  (increase  in  performance)  of 
24.5  mm  from  the  first  to  fourth  attempt  at  LF1,  consistent  with  learning  during  training. 
This  difference  was  statistically  significant  (z  =  -1.98,  p  =  0.05).  However,  as  will  be 
discussed  in  Section  4.5.2,  the  LF1  practice  is  much  simpler  than  the  LF6  practice,  and  is 
therefore  not  sufficient  preparation  for  LF6  qualification. 

The  mean  LF6  simulator  scores  for  all  participants,  as  well  as  those  for  each  group  in  the 
simulator  are  given  in  Table  11  (Appendix  F).  For  the  experimental  group,  there  was  an 
increase  of  12.9  points  from  the  first  to  third  attempt.  This  change  was  not  statistically 
significant  (z  =  -1.17,  p  =  0.24).  On  this  basis,  there  was  no  evidence  that  learning  had 
occurred  during  the  LF6  simulator  training. 


4.  Discussion 


This  section  discusses  the  results  from  the  study  in  the  context  of  the  study  aims. 
Implications  of  the  findings  for  marksmanship  training  are  also  outlined  and  discussed,  as 
well  as  options  for  future  research. 
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4.1  Training  effectiveness 

The  first  aim  of  the  study  was  to  measure  the  effect  of  additional  LF6  simulator  training  on 
live-fire  qualification.  The  primary  measure  of  training  effectiveness  used  in  this  study 
was  the  difference  between  LF6  qualification  rates  between  the  two  groups.  Using  this 
measure,  the  results  indicated  that  the  group  that  conducted  additional  training  in  the 
simulator  qualified  faster  than  the  group  that  went  straight  to  the  live-fire  range. 
However,  on  the  basis  of  mean  LF6  qualification  scores,  any  difference  between  the  two 
groups  was  too  small  to  be  detected.  Overall,  it  can  be  concluded  that  there  was  a  small, 
positive  transfer  of  training  from  the  simulator  to  live-fire  for  the  LF6  practice  (i.e.  Level  3 
of  the  Kirkpatrick  framework).  A  comparison  of  the  pass  rates  for  the  two  groups 
indicated  that  the  three  LF6  practices  in  the  simulator  had  roughly  the  same  benefit  as  one 
LF6  practice  on  the  live-fire  range.  Overall,  these  findings  partially  support  the  study 
hypothesis  and  are  broadly  consistent  with  previous  studies  examining  the  training 
effectiveness  of  marksmanship  simulators  [1,  6,  14,  16-18],  That  is,  there  is  evidence  for 
limited  transfer  of  training  from  the  simulator  to  live-fire,  and  no  evidence  for  negative 
training. 

Ideally,  all  trainees  would  qualify  first  or  second  attempt,  however  neither  group  came 
close  to  achieving  this  outcome.  It  took  ten  attempts  and  two  and  a  half  days  before  all 
trainees  had  qualified.  Based  on  this  outcome,  the  training  conducted  by  both  groups  is 
not  particularly  effective.  Prior  to  the  first  three  qualification  attempts,  the  two  groups 
received  different  training,  both  in  terms  of  the  total  number  of  rounds  fired,  and  the  mix 
of  simulator  and  live-fire  training  (see  Table  6).  The  three  additional  LF6  simulator 
practices  conducted  by  the  experimental  group  equate  to  67%  more  rounds.  Given  that  the 
experimental  group  performed  only  slightly  better  than  the  control  group  over  the  first 
three  LF6  qualification  attempts,  this  additional  training  represents  a  poor  return  on 
investment  for  Army.  Potential  reasons  for  these  findings,  together  with  some 
recommendations  regarding  areas  for  improvement  to  the  marksmanship  training,  are 
discussed  in  Section  4.5. 


4.2  Predictors  of  LF6  qualification 

The  second  aim  of  this  study  was  to  investigate  predictors  of  LF6  qualification.  Ideally, 
training  staff  would  be  able  to  use  simulator  scores  to  decide  whether  trainees  were  ready 
to  proceed  to  LF6  qualification,  and  avoid  wasting  live  ammunition  on  trainees  who  were 
likely  to  take  multiple  attempts  to  pass.  However,  the  correlation  between  simulator  and 
live-fire  performance  (r  =  0.4)  was  too  small  to  achieve  this  outcome;  it  was  not  possible  to 
set  a  cut-off  score  in  the  simulator  which  would  allow  reliable  discrimination  between 
those  who  were  likely  to  qualify  and  those  who  were  not.  An  additional  implication  of  this 
finding  is  that  the  simulator  should  not  be  used  to  replace  live-fire  for  marksmanship 
assessment. 

Similar  outcomes  have  been  found  in  other  studies  which  compared  firer  performance  in 
the  two  environments,  with  correlation  coefficients  generally  low  to  moderate  [3,  8, 10-11]. 
The  observed  magnitude  of  r  is  likely  to  be  due  to  three  factors:  (1)  the  particular 
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marksmanship  task  being  conducted  (2)  variance  in  firer  performance  across  trials  and  (3) 
differences  between  the  simulator  and  live-fire  environments.  In  this  study,  a  repeated 
measures  method  was  employed  with  simulator  and  live-fire  scores  being  averaged  over 
three  attempts.  This  will  reduce  the  effect  of  firer  variance  across  trials  and  should  result  in 
higher  values  of  r  than  for  a  single  measurement.  By  way  of  comparison,  the  correlations 
between  LF6  practices  conducted  on  the  live-fire  range  (e.g.  correlation  between  live-fire 
1st  attempt  at  LF6  and  live-fire  2nd  attempt  at  LF6)  and  those  between  LF6  practices 
conducted  in  the  simulator  (e.g.  correlation  between  simulator  1st  attempt  at  LF6  and 
simulator  2nd  attempt  at  LF6)  were  all  higher  than  0.422.  Consequently,  the  correlation  (r  = 
0.4)  observed  between  simulator  and  live-fire  LF6  scores  is  unlikely  to  be  due  to  firer 
variance,  and  is  more  likely  to  reflect  differences  between  the  simulator  and  live-fire 
environments  for  the  LF6  task  (e.g.  target  imagery,  weapon  recoil).  These  differences  are 
discussed  further  in  Section  4.5.1.  The  implication  here  is  that  the  LF6  task  in  the  simulator 
is  similar,  but  not  equivalent,  to  the  LF6  task  on  the  live-fire  range. 

Interestingly,  self-ratings  of  marksmanship  ability  showed  greater  potential  than  simulator 
scores  in  predicting  LF6  qualification.  While  it  is  not  recommended  that  trainees  be 
allowed  to  proceed  straight  to  qualification  on  the  basis  of  self-report  data,  such  data  could 
be  used  on  future  courses  to  identify  trainees  requiring  additional  assistance  in  the  early 
stages  of  training. 


4.3  Comparison  with  2009  study  outcomes 

The  third  aim  of  the  study  was  to  compare  the  outcomes  with  those  of  the  2009  course  [17]. 
At  face-value,  the  outcomes  of  the  current  study  suggest  that  the  benefit  of  the  additional 
simulator  training  was  less  in  the  current  study  than  in  the  2009  study.  However,  as  shown 
in  Appendix  E,  these  differences  can  be  explained  by  differences  in  the  baseline 
marksmanship  ability  levels  of  trainees  between  the  two  studies  (prior  to  conducting  LF6 
training  in  the  simulator),  and  the  use  of  pass  rates  as  a  measure  of  effectiveness  (in 
conjunction  with  small  sample  sizes). 

Differences  in  baseline  marksmanship  ability  between  the  2009  and  2010  studies  could  be 
due  to  differences  in  trainee  characteristics23,  but  are  more  likely  to  be  a  result  of 
differences  in  the  overall  training  method  used  in  the  two  studies.  Potential  differences 
include  the  amount  of  marksmanship  coaching,  the  specific  practices  conducted  (prior  to 
the  additional  LF6  simulator  training),  and  the  amount  of  live-fire  training.  Without 
knowing  the  specific  training  conducted  in  the  2009  study,  it  is  not  possible  to  speculate 
further. 


22  Live-fire:  r  =  0.65,  p  <  0.001  (1st  vs.  2nd  attempt);  r  =  0.68,  p  <  0.001  (1st  vs  3rd  attempt);  r  =  0.65,  p  < 
0.001  (2nd  vs  3rd  attempt).  Simulator:  r  =  0.61,  p  <  0.001  (1st  vs  2nd  attempt);  r  =  0.45,  p  <  0.01  (1st  vs  3rd 
attempt);  r  =  0.50,  p  <  0.01  (2nd  vs  3rd  attempt). 

23  There  is  no  demographic  data  available  for  trainees  in  the  2009  study  so  we  cannot  compare 
demographic  variables  across  the  two  studies. 
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4.4  Reverse  transfer  of  training 

As  shown  in  Section  3.4.1  and  Appendix  F,  there  was  no  evidence  to  suggest  that 
conducting  LF6  on  the  live-fire  range  had  any  impact  on  subsequent  LF6  performance  in 
the  simulator.  Given  that  differences  between  the  simulator  and  live-fire  ranges  may  have 
limited  the  transfer  of  training  from  the  simulator  to  live-fire  (Section  4.5.1),  it  is  likely  that 
these  differences  would  also  impact  on  the  reverse  transfer  of  training.  Furthermore, 
trainees  in  the  experimental  group  conducted  the  LF6  training  in  the  simulator  as 
preparation  for  qualification;  trainees  in  the  control  group  conducted  the  simulator 
training  only  to  ensure  both  groups  received  equal  training  after  having  already 
conducted  three  attempts  at  qualification.  The  50%  of  trainees  in  the  control  group  that 
failed  to  qualify  after  three  attempts  may  have  some  incentive  to  make  use  of  this  training 
opportunity;  the  other  50%  of  trainees  that  had  already  qualified  may  have  had  limited 
incentive  to  perform  well  in  the  simulator. 


4.5  Potential  reasons  for  study  findings  and  possible  future  training 
options 

As  discussed  in  Section  4.1,  the  additional  simulator  training  conducted  by  the 
experimental  group,  as  well  as  the  overall  training  conducted  by  both  groups,  failed  to 
lead  to  satisfactory  qualification  rates  (e.g.  qualify  first  or  second  attempt).  The  following 
sections  outline  possible  reasons  for  these  findings,  together  with  a  number  of  options  for 
future  studies  aimed  at  improving  training  outcomes24.  While  these  options  may  appear  to 
require  additional  training  and  resources,  if  they  proved  effective,  they  would  result  in 
less  time  and  resources  being  spent  qualifying  on  LF6. 

4.5.1  Limitations  of  the  simulator  environment 

One  reason  for  the  limited  benefit  resulting  from  the  additional  simulator  training  could 
be  the  difference  between  the  target  characteristics  in  the  simulator  compared  with  those 
on  the  live-fire  range.  It  has  been  previously  reported  that  the  target  brightness  and 
contrast  in  the  simulator  are  significantly  lower  than  on  the  live-fire  range  under  daytime 
ambient  conditions  [7-8].  This  has  implications  for  firers  when  attempting  to  acquire  and 
engage  targets,  particularly  at  longer  ranges,  with  'snap'  serials  and  when  using  iron 
sights.  If  the  firer  has  difficulty  acquiring  the  target,  it  is  likely  that  this  will  limit  the 
amount  of  training  benefit  possible.  This  is  consistent  with  the  lower  scores  in  the 
simulator  compared  with  live-fire  (Appendix  C),  the  poorer  performance  at  200  m  and 
300  m  serials  (Appendix  C)  and  the  moderate  correlation  between  simulator  and  live-fire 
scores  (Appendix  D).  It  is  also  consistent  with  participants'  comments  (Section  3.4.2  and 
Appendix  G)  and  findings  from  previous  studies  [8]25. 


24  Several  of  these  options  were  subsequently  investigated  as  part  of  a  trial  on  a  subsequent  course; 
the  results  of  this  trial  are  reported  elsewhere  [26]. 

25  DSTO  research  has  also  found  that  the  hit-detection  accuracy  in  marksmanship  simulators  is  less 
than  on  the  live-fire  range  [12].  However,  the  level  of  accuracy  is  high  relative  to  the  size  of  the 
targets  employed  in  LF6  and  is  therefore  not  likely  to  be  a  significant  problem. 
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The  simplest  method  for  improving  target  image  brightness  and  contrast  would  be  to 
change  the  appearance  of  the  target;  for  example,  by  using  a  plain  white  or  orange  target. 
Most  marksmanship  simulators  have  an  authoring  function  which  would  readily  allow 
such  a  solution  to  be  implemented26. 

As  noted  in  Appendix  G,  the  lower  weapon  recoil  in  the  simulator  compared  with  the  real 
weapon  may  be  an  additional  factor  that  limits  the  effectiveness  of  training  in  the 
simulator.  While  the  lower  recoil  is  likely  to  make  it  easier  to  engage  targets  in  the 
simulator,  this  may  actually  provide  a  negative  training  effect  when  using  the  real 
weapon.  There  is  no  obvious  safe  solution  to  address  this  issue.  Should  the  improvements 
to  the  target  imagery  result  in  better  training  outcomes  then  it  may  not  be  necessary  to 
modify  the  recoil. 

4.5.2  Training  progression 

Prior  to  the  LF6  practices  in  the  simulator,  the  trainees  conducted  several  LF1  practices  on 
the  live-fire  range,  and  conducted  the  LF4  and  LF5  practices  once  each  in  the  simulator 
(Table  2).  The  LF1  practice  is  a  100  m  grouping  practice  fired  from  the  prone  unsupported 
position  and  is  consequently  much  simpler  than  LF6,  which  involves  engaging  targets  up 
to  300  m  from  various  firing  positions.  There  are  some  similarities  between  the  serials  in 
LF6  and  those  in  LF4  and  LF5,  however,  these  practices  were  conducted  only  once  in  the 
simulator.  If  the  LF6  practices  conducted  in  the  simulator  are  included,  the  training  did 
provide  a  systematic  progression  towards  live  LF6  qualification  (i.e.  the  crawl-walk-run 
approach  [27]).  However  this  appears  to  have  been  confounded  by  the  limitations  in  the 
simulator  environment  identified  above;  overall,  there  was  a  lack  of  live-fire  training  at 
longer  ranges.  The  net  impact  is  that  the  less  skilled  firers  (i.e.  those  that  failed  to  pass  any 
of  their  first  three  qualification  attempts)  had  particular  difficulty  at  engaging  targets  at 
200  m  and  300  m  (Appendix  C).  Consequently,  the  training  did  not  provide  the  necessary 
progression  from  simple  to  more  difficult  tasks  required  as  preparation  for  LF6 
qualification. 

The  LF6  practice  contains  two  serials  at  300  m,  three  serials  at  200  m  and  three  serials  at 
100  m;  the  practice  also  includes  four  snap  serials.  The  LF4  practice  has  serials  at  100  m 
and  200  m;  there  are  no  serials  at  300  m  and  no  snap  serials.  The  LF5  practice  involves  only 
moving  targets  at  100  m.  Given  that  trainees  had  the  most  problems  with  targets  at  200  m 
and  300  m  during  qualification,  then  the  LF4  and  LF5  practices  alone  may  not  provide 
sufficient  preparation  for  the  LF6  practice.  This  view  is  supported  by  the  fact  that  trainees 
performed  reasonably  well  at  LF4  and  LF5  in  the  simulator  prior  to  the  study  (pass  rates 
were  71%  and  84%,  respectively27).  Consequently,  future  courses  could  focus  on  engaging 
targets  at  200  m  and  300  m.  This  could  involve  a  modified  LF6  practice  where  trainees  start 
with  grouping  serials  from  the  prone  unsupported  position  with  no  timing  constraints 
before  progressing  to  rapid  and  snap  serials,  as  well  as  firing  from  other  positions.  The  LF3 


26  DSTO  has  subsequently  received  approval  from  the  manufacturer  of  the  simulator  to  implement 
the  proposed  solution.  Without  such  approval,  it  would  have  been  necessary  to  pay  the 
manufacturer  to  perform  the  changes. 

27  This  data  was  provided  to  the  study  team  by  the  course  instructional  staff. 
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practice  (which  involves  deliberate  and  rapid  serials  from  the  prone  unsupported  position 
at  ranges  of  200  m  and  300  m)  may  meet  some  of  these  requirements. 

4.5.3  M4  and  iron  sights 

The  difficulty  in  using  iron  sights  is  another  factor  that  could  explain  the  study  findings. 
As  noted  previously,  most  of  the  participants  had  limited  experience  with  the  M4  weapon 
and  iron  sights  and  were  more  experienced  with  optical  sighting  systems.  Use  of  the  iron 
sight  requires  different  aiming  and  sighting  techniques  to  those  employed  with  optical 
sighting  systems.  As  noted  by  the  participants  (Section  3.4.2,  Appendix  G),  it  is  generally 
accepted  that  it  is  easier  to  engage  targets  with  optical  sights  compared  with  iron  sights 
(especially  at  longer  ranges).  In  particular,  for  static  targets  such  as  those  on  live-fire 
ranges,  acquisition  times  are  likely  to  be  shorter  with  an  optical  sight.  In  addition,  the 
current  basic  marksmanship  practices  in  the  Australian  Army  (i.e.  LF1  to  LF6)  have  been 
designed  primarily  for  the  Steyr  optical  sighting  system. 

The  M4  also  requires  different  holding  techniques  to  the  Steyr,  and  the  position  and  hold 
of  the  weapon  is  changed  slightly  for  different  target  ranges  and  firing  positions.  Although 
trainees  received  lessons  on  holding,  aiming  and  firing  the  M4  as  part  of  the  course, 
trainees  had  a  limited  chance  to  become  familiar  with  the  M4  prior  to  attempting  LF6. 

The  trainee's  lack  of  experience  with  the  M4  weapon  system  and  iron  sight  could 
potentially  be  addressed  by  conducting  dry  firing,  as  well  as  implementing  some  practices 
specific  to  this  weapon  system.  Army  doctrine  notes  that  dry  firing  is  an  important 
component  of  marksmanship  training  [13],  and  would  be  particularly  useful  for  greater 
familiarisation  with  the  M4  and  iron  sights.  Additional  options  could  include  commencing 
training  by  conducting  grouping  practices  at  25  m  first,  then  at  100  m  in  order  to  develop 
confidence  by  conducting  a  simple  task  in  the  early  stages  of  the  training.  Once  this  has 
occurred,  training  could  focus  on  conducting  practices  that  progress  to  LF6  as  outlined  in 
Section  4.5.2  above. 

Given  that  trainees  are  generally  used  to  firing  with  optical  sights,  another  option  is  to 
commence  M4  training  using  an  optical  sight.  After  trainees  had  become  sufficiently 
familiar  with  this  system,  they  could  then  remove  the  optical  sight  and  use  the  iron  sight. 
Alternatively,  the  requirement  to  qualify  using  iron  sights  could  be  changed  to  allow 
qualification  using  optical  sights;  however,  decision-makers  would  need  to  weigh  up  the 
impact  on  marksmanship  capability  levels  before  implementing  such  a  policy  change. 

4.5.4  Coaching  and  feedback 

Army  doctrine  notes  that  effective  coaching  at  all  stages  of  training  is  essential  to  improve 
shooting  standards  [13];  consequently,  the  absence  of  individual  coaching  is  likely  to  be  a 
factor  in  explaining  the  study  findings.  One  reason  for  the  poor  performance  at  LF6 
qualification  may  be  that  trainees  were  not  applying  the  basic  marksmanship  principles 
correctly  during  the  study.  Alternatively,  they  may  simply  have  not  been  using  the  iron 
sight  correctly,  due  to  a  lack  of  experience.  Either  way,  the  only  way  to  address  these 
issues  is  with  marksmanship  coaching. 
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Fall-of-shot  feedback  was  available  in  the  simulator  and  live-fire  range;  however  based  on 
the  LF6  qualification  outcomes,  this  does  not  appear  to  have  been  of  benefit  to  the  trainees. 
An  inspection  of  the  fall-of-shot  pattern  from  the  live-fire  qualification  data  showed  that 
there  were  a  few  instances  where  firers  appeared  to  be  systematically  aiming  off  target.  As 
this  was  a  qualification  task,  this  information  was  only  available  to  trainees  at  the  end  of 
the  practice.  However,  trainees  did  have  the  opportunity  to  adjust  their  point-of-aim  using 
the  sighting  rounds  fired  in  Serial  1.  Overall,  low  scores  appeared  to  correspond  to  a 
significant  spread  in  shot  location,  consistent  with  poor  application  of  marksmanship 
techniques;  this  could  not  have  been  addressed  without  coaching. 

When  conducting  LF6  in  the  simulator,  trainees  were  able  to  see  their  fall-of-shot  after 
each  serial,  but  were  not  provided  with  any  corrective  action  from  training  staff  based  on 
this  information.  Research  has  shown  that  trainee  learning  is  facilitated  when  they  receive 
specific  feedback  about  processes  (e.g.  marksmanship  technique)  and  outcomes  (e.g.  LF6 
scores,  fall-of-shot  pattern)  associated  with  the  task  being  conducted  [28],  In  this  trial,  the 
feedback  provided  to  trainees  was  mostly  outcome-based.  Given  that  the  resultant  effect  of 
additional  simulator  training  was  limited,  it  is  possible  that  a  greater  degree  of  coaching 
would  have  resulted  in  greater  training  effectiveness  in  terms  of  subsequent  LF6 
qualification.  It  has  previously  been  found  that  transfer  of  marksmanship  skills  from  the 
simulator  to  live-fire  is  greater  when  coaching  is  employed  [14],  However,  in  the  specific 
case  of  conducting  LF6  with  iron  sights,  the  limitations  with  the  simulator  target  imagery 
may  need  to  be  addressed  to  maximise  the  benefit  of  coaching. 

4.5.5  Tailoring  training  to  trainee  requirements 

In  any  training  program  where  the  aim  is  to  ensure  all  personnel  pass,  training  throughput 
will  be  limited  by  the  slowest  trainees.  Out  of  the  thirty  six  participants,  nine  passed  one  of 
their  first  four  attempts,  and  eleven  failed  to  pass  any  of  their  first  four  attempts.  Targeting 
these  two  groups  is  where  the  greatest  efficiencies  can  be  achieved  because  these  trainees 
consume  the  most  time  and  resources.  Individuals  requiring  additional  training  should  be 
identified  early  in  (or  prior  to)  the  course;  they  could  then  receive  additional  training 
and/or  coaching.  If  practical,  the  more  skilled  firers  could  progress  more  quickly  to  LF6 
without  this  additional  assistance28. 

As  noted  in  Section  4.2,  self-ratings  of  marksmanship  ability  were  reasonable  predictors  of 
LF6  qualification.  If  additional  data  was  collected  prior  to  the  course  (e.g.  trainees  most 
recent  live-fire  scores),  this  information  could  help  to  (1)  identify  trainees  likely  to  have 
difficulty  qualifying  at  LF6  and  (2)  assist  with  determining  resource  requirements  for  the 
course.  In  theory,  trainees  should  record  such  information  in  a  shooters  log. 


28  The  authors  acknowledge  that  there  needs  to  be  an  appropriate  balance  between  providing 
adequate  training  and  not  'spoon-feeding'  the  trainees  (i.e.,  providing  trainees  with  too  much 
feedback/ coaching).  It  is  important  that  the  more  skilled  firers  are  still  given  enough  feedback  in 
order  to  support  their  progress  during  training. 
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4.6  Study  limitations 

The  main  limitation  in  this  study  was  the  small  sample  size,  with  only  18  participants  in 
each  group.  In  addition,  the  course  is  only  run  annually,  which  means  there  is  infrequent 
opportunity  to  build  upon  the  outcomes  of  the  current  study  and  examine  the  training 
options  outlined  in  Section  4.5  above. 

The  other  main  limitation  was  the  requirement  to  fit  the  study  design  within  the 
constraints  of  the  training  course,  in  particular,  the  requirement  for  both  experimental  and 
control  group  to  conduct  the  LF6  simulator  training.  While  this  had  minimal  impact  on  the 
study  outcomes,  it  constrained  the  options  for  the  study  design. 


5.  Conclusion  and  recommendations 


The  aim  of  the  study  was  to  measure  the  effect  of  additional  LF6  simulator  training  on 
live-fire  qualification.  The  main  finding  of  this  study  was  that  additional  LF6  training  in 
the  simulator  provided  a  small,  positive  benefit  in  terms  of  subsequent  LF6  qualification. 
Specifically,  three  LF6  practices  in  the  simulator  were  of  similar  effectiveness  to  one  LF6 
practice  on  the  live-fire  range;  this  outcome  partially  supports  the  study  hypothesis.  The 
findings  are  broadly  consistent  with  previous  studies,  which  have  found  evidence  of 
positive  transfer  of  training  from  the  simulator  to  live-fire. 

Overall,  however,  the  additional  simulator  training  provided  a  poor  return  on  investment 
in  terms  of  the  subsequent  qualification  rates,  with  the  majority  of  trainees  in  both  groups 
requiring  numerous  attempts  to  qualify.  Consequently,  the  simulator  training,  in 
conjunction  with  the  overall  training  method  used  on  the  course  was  not  satisfactory.  The 
most  likely  reasons  for  this  outcome  include  limitations  in  the  simulator  target  imagery, 
the  lack  of  experience  of  the  trainees  with  the  M4  weapon  and  iron  sights,  and  the  limited 
coaching/  feedback  provided  to  trainees  during  the  study.  The  clear  message  here  is  that 
many  factors  contribute  to  effective  training  beyond  the  utility  of  the  specific  training 
environment  (i.e.  simulation  or  live-fire).  Furthermore,  unless  consideration  is  given  to 
these  factors,  it  is  unlikely  that  satisfactory  training  outcomes  will  be  achieved  and 
questions  regarding  the  effectiveness  of  the  simulator,  and  the  optimum  blend  of  live-fire 
and  simulator  training  are  secondary. 

Additional  studies  will  therefore  be  required  in  order  to  develop  a  more  effective 
marksmanship  training  methodology  for  live-fire  LF6  qualification  with  the  M4/iron 
sights  which  uses  a  combination  of  simulation  and  live-fire.  The  optimum  solution  is  likely 
to  be  one  that  is  tailored  to  accommodate  trainees  with  a  wide  range  of  marksmanship 
ability  and  ensures  trainees  follow  a  logical  progression  from  simpler  to  more  difficult 
tasks,  in  line  with  marksmanship  doctrine  [13].  Accordingly,  it  is  recommended  that  the 
following  options  are  investigated  in  future  studies: 

•  Improving  the  brightness  and  contrast  of  the  target  images  in  the  simulator  and 
evaluating  any  impact  on  subsequent  training  effectiveness  (Section  4.5.1). 
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•  Designing,  implementing  and  evaluating  marksmanship  practices  aimed  at  improving 
trainee  familiarisation  and  confidence  with  the  M4  weapon  system  and  iron  sights, 
which  follow  a  logical  progression  from  simpler  to  more  difficult  tasks  (Section  4.5.2 
and  4.5.3).  This  could  also  include  (initially)  using  the  M4  with  an  optical  sight. 

•  Providing  individual  marksmanship  coaching  to  trainees  to  ensure  correct  application 
of  marksmanship  principles  and  M4  weapon  handling  (Section  4.5.4). 

•  Collecting  data  on  trainee  marksmanship  ability  prior  to  training  courses  to  identify 
trainees  likely  to  require  additional  assistance  (Section  4.5.5).  This  could  be  done  using 
a  simple  self-assessment  of  ability  and/or  objective  data  on  each  trainee's 
marksmanship  performance  (e.g.  previous  LF6  scores  from  the  shooters  log). 
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Appendix  A:  Description  of  marksmanship  practices 

The  range  practices  for  basic  marksmanship  proficiency  (in  order  of  increasing  difficulty) 
are: 

a.  LF1:  the  grouping  practice  (100  m) 

b.  LF2:  The  Zeroing  Practice  (100  m) 

c.  LF3:  The  Application  of  Fire  Prone  Practice  (200  to  300  m) 

d.  LF4:  the  Application  of  Fire  (Other  Positions)  Practice 

e.  LF5:  The  Moving  Target  Practice 

f.  LF6:  The  Application  of  Fire  Practice. 

LF1  -  LF5  are  described  briefly  below;  LF6  is  described  in  detail  in  Table  7.  The  targets 
used  for  the  LF6  practice  are  shown  in  Figure  2. 

•  LF1:  the  grouping  practice  100  m.  This  involves  firing  4x5  round  groups  from  the 
prone  unsupported  position  at  a  static  target  at  100  m.  The  aim  of  the  practice  is  to 
teach  shot  grouping  ability,  i.e.  precision. 

•  LF2:  the  zeroing  practice  100  m.  This  involves  firing  4x5  round  groups  from  the 
prone  unsupported  position  at  a  static  target  at  100  m.  The  aim  of  the  practice  is  to 
confirm  weapon  zero  and  make  adjustments  to  the  weapon  sight  as  required. 

•  LF3:  the  application  of  fire  prone  practice  200  to  300  m.  This  involves  engaging 
targets  at  distances  of  200  m  and  300  m  from  the  prone  unsupported  position, 
using  the  deliberate  and  rapid  techniques.  The  aim  of  the  practice  is  to  teach  the 
application  of  fire  to  the  maximum  effective  range  of  the  weapon. 

•  LF4:  the  application  of  fire  (other  positions)  practice-  100  to  200  m.  This  involves 
engaging  targets  at  distances  of  100  m  and  200  m  from  sitting,  kneeling  or 
squatting  and  standing  positions  both  supported  and  unsupported,  using  the 
deliberate  and  rapid  techniques.  The  aim  of  the  practice  is  teach  the  application  of 
fire  from  different  positions. 

•  LF5:  the  moving  target  practice.  This  involves  engaging  moving  targets  at  100  m; 
with  targets  moving  left  or  right,  and  at  different  speeds.  The  aim  of  the  practice  is 
to  teach  the  appropriate  technique  for  engaging  moving  targets  at  ranges  up  to 
100  m. 

•  LF6:  the  application  of  fire  practice.  This  involves  engaging  static  targets  at 
distances  of  100  m,  200  m  and  300  m  and  moving  targets  at  100  m  from  all 
conventional  firing  positions,  using  deliberate,  rapid  and  snap  techniques. 


UNCLASSIFIED 


31 


DSTO-TR-2950 


UNCLASSIFIED 


32 


UNCLASSIFIED 


UNCLASSIFIED 


DSTO-TR-2950 


Table  7.  The  LF6  practice.  Scoring  is  five  points  per  hit  on  target.  HPS  =  highest  possible  score. 


Serial 

Type 

Range (m) 

Rounds 

Target  Type 

Firing  Position 

Description  and  Timings 

HPS 

1 

Deliberate 

Timed 

300 

8 

Figure  11 

Standing 
Supported 
(Weapon  Pit) 

Two  sighting  rounds  (in  firers  own  time)  are  fired  at  a 
Figure  11  target  followed  by  six  rounds  to  score  in 
two  minutes. 

30 

2 

Snap 

300 

6 

Figure  11 

Prone 

Unsupported 

One  round  is  fired  at  each  of  six  exposures  of  a  Figure 
11  target.  Targets  are  exposed  for  3  seconds  with  an 
interval  of  5  to  10  seconds  between  exposures. 

30 

3 

Rapid 

200 

6 

Figure  12 

Sitting 

Supported 

6  rounds  are  fired  at  one  24  second  exposure  of  a 
Figure  12  target. 

30 

4 

Snap 

200 

6 

Figure  12 

Kneeling  or 

Squatting 

Supported 

1  round  is  fired  at  each  of  six  exposures  of  a  Figure  12 
target.  Targets  are  exposed  for  3  seconds  with  an 
interval  of  5  to  10  seconds  between  exposures. 

30 

5 

Rapid 

200 

6 

Figure  12 

Sitting 

Unsupported 

6  rounds  are  fired  at  one  30  second  exposure  of  a 
Figure  12  target. 

30 

6 

Snap 

100 

6 

Figure  11 

Kneeling  or 

Squatting 

Unsupported 

1  round  is  fired  at  each  of  six  exposures  of  a  Figure  11 
target.  Targets  are  exposed  for  3  seconds  with  an 
interval  of  5  to  10  seconds  between  exposures. 

30 

7 

Snap 

100 

6 

Figure  11 

Standing 
Unsupported 
then  Kneeling  or 
Squatting 
Unsupported 

1  round  is  fired  at  each  of  six  exposures  of  a  Figure  11 
target.  Targets  come  in  3  sets  of  2  targets;  in  each  set 
targets  are  engaged  from  a  different  position.  Target 
exposures  are  for  3  seconds,  with  a  3  second  interval 
between  sets. 

30 

8 

Moving 

Target 

100 

15 

Figure  11 

Kneeling  or 

Squatting 

Supported 

A  maximum  of  3  rounds  are  fired  at  each  of  5 
exposures  of  a  moving  Figure  11  target.  Target  speeds 
are  random  (1,  2  or  3  ms4).  Targets  move  either  right 
to  left  or  left  to  right  over  10  metres. 

25 

Total 

59 

235 
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Appendix  B:  Demographic  questionnaire 

Rank .  Age . 

First  Name .  Surname . 

Previous  Unit .  Years  of  ADF  Service . 

Branch  of  Service .  Corp/ Mustering . 

1.  How  many  times  have  you  fired  a  rifle  in  the  past  12  months  (including  WTSS  and  live-fire)? 

2.  Prior  to  the  course,  how  would  you  have  rated  your  marksmanship  skill  level? 

I  I  Below  average  (do  not  consistently  pass  Annual  Weapons  Test  first  time) 

I  I  Average  (usually  pass  Annual  Weapons  Test  first  time) 

I  I  Above  average  (always  pass  Annual  Weapons  Test  first  time  and  score  well  below  pass  mark) 

3.  How  many  times  have  you  conducted  the  LF6  practice  in  your  career? 

4.  When  was  last  time  you  shot  LF6?  Specify  month  and  year. 

5.  Prior  to  this  course,  have  you  ever  used  the  M4  weapon?  QYes  I  I  No 

a.  If  yes,  how  much  experience  (in  months  or  years)  have  you  had  with  the  M4  weapon? 

6.  Prior  to  this  course,  have  you  ever  used  a  weapon  with  iron  sights?  Q  Yes  Q  No 

a.  If  yes,  please  specify  weapon  system . 

b.  How  much  experience  have  you  had  with  that  weapon  (in  months  or  years)? 

7.  Based  on  your  current  level  of  training,  how  would  you  rate  your  confidence  in  using  the  M4? 

□  Low  O  Moderate  Q  High 

8.  How  confident  are  you  of  passing  LF6  first  time  with  the  M4  weapon? 

□  Low  Q  Moderate  Q  High 

9.  Do  you  wear  corrective  glasses/ eyewear?  I  I  Yes  I  I  No 

10.  Do  you  have  20/20  vision?  I  I  Yes  I  I  No 

11.  How  would  you  rate  your  general  health  today?  (please  tick) 

I  I  Well  Below  Average 

□  Below  Average 
I  I  Average 

I  I  Good 
I  I  Excellent 

12.  Are  there  any  factors  (e.g.  illness,  injury)  that  you  believe  may  affect  your  marksmanship  performance 
this  trial? 

O  Yes  A  please  describe: . 

□  No  . 
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Appendix  C:  Analysis  of  marksmanship  performance 
data  to  identify  potential  training  improvements 

The  following  sections  present  the  results  of  an  analysis  of  marksmanship  performance 
data  from  the  study  that  was  conducted  to  help  identify  potential  training  improvements. 

C.l.  LF6  serial  scores  based  on  live-fire  qualification  rates 

The  mean  scores  for  each  live-fire  LF6  serial  are  shown  in  Figure  3;  participants  were 
classified  according  to  the  number  of  times  they  passed  their  first  three  attempts  at  LF6  on 
the  live-fire  range  (i.e.  none,  one,  two/three29).  Values  are  shown  as  a  percentage  of  the 
maximum  possible  score  achievable  for  each  serial.  Participants  performed  better  at  100  m 
serials  than  at  200  m  and  300  m.  The  differences  between  firers  that  passed  two  or  three 
attempts  and  firers  that  did  not  pass  on  any  attempt  are  greatest  for  Serial  1  (mean 
difference  =  7.9,  t  =  3.50,  p  =  0.002,  d  =  1.41),  Serial  3  (mean  difference  =  10.3,  t  =  6.03,  p  < 
0.001,  d  =  2.36),  Serial  4  (mean  difference  =  9.5,  t  =  4.41,  p  <  0.001,  d  =  1.72)  and  Serial  5 
(mean  difference  =  8.3,  t  =  5.48,  p  <  0.001,  d  =  2.11).  All  participants  performed  poorly  at 
Serial  2;  a  snap  serial  at  300  m.  While  an  improvement  in  firer  performance  for  all  serials 
would  lead  to  faster  LF6  qualification,  these  results  suggest  that  marksmanship  training 
should  focus  on  tasks  similar  to  Serials  1,  3,  4  and  5,  where  the  poorer  firers  struggled  the 
most;  this  is  discussed  further  in  Section  4.5.2. 
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Figure  3.  Mean  score  for  each  serial  for  the  first  three  attempts  at  LF6  on  the  live-fire  range  as  a 
percentage  of  the  maximum  possible  score.  Participants  are  classified  according  to  the 
number  of  times  they  achieved  a  qualification  score  (i.e.  >  165  points). 


29  Scores  for  firers  who  passed  two  or  three  times  in  their  first  three  qualification  attempts  were 
combined  due  to  small  sample  sizes. 
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C.2.  Differences  between  simulator  and  live-fire  scores 

The  mean  scores  for  all  participants  for  serials  at  100  m,  200  m  and  300  m,  and  for  the  LF6 
practice  in  the  simulator  and  on  the  live-fire  range  are  shown  in  Table  8.  The  mean  LF6 
scores  on  the  live-fire  range  were  higher  than  those  in  the  simulator,  but  the  difference  was 
not  statistically  significant  (t  =  1.72,  p  =  0.09).  The  differences  between  simulator  and  live- 
fire  mean  scores  increase  with  target  range.  These  differences  were  statistically  significant 
for  300  m  (t  =  2.57,  p  =  0.01,  d  =  1.12)  and  200  m  (z  =  -1.96,  p  =0.05)  but  not  at  100  m  (t  = 
1.21,  p  =  0.24).  Overall,  the  data  suggests  that  participants  found  it  easier  to  hit  targets  at 
longer  ranges  on  the  live-fire  range  than  they  did  in  the  simulator30.  This  is  supported  by 
participants'  feedback  (see  Appendix  G),  which  indicated  that  they  had  difficulty  seeing 
the  LF6  targets  in  the  simulator,  particularly  when  targets  were  at  200  m  and  300  m. 
Consequently,  more  effective  training  could  potentially  be  achieved  by  improving  the 
target  visibility  in  the  simulator.  This  is  discussed  further  in  Section  4.5.1. 


Table  8.  Mean  scores  for  100  m,  200  m,  and  300  m  serials  in  the  simulator  and  live-fire  range. 
Numbers  in  brackets  are  one  standard  deviation. 


Serials 

Range (m) 

Simulator 

Live-fire 

range 

Difference 

6,  7  and  8 

100 

22.8 

23.5 

0.7 

(2.6) 

(2.7) 

3,  4  and  5 

200 

13.8 

15.4 

1.6 

(4.3) 

(5.0) 

1  and  2 

300 

13.1 

15.6 

2.5 

(4.5) 

(5.0) 

All 

100,  200,  300 

137.5 

148.0 

10.5 

30  This  analysis  is  based  on  all  participants'  scores  so  does  not  take  into  account  the  order  in  which 
trainees  shot  in  the  simulator  and  on  the  live  range.  However,  similar  results  were  obtained  when 
the  analysis  was  conducted  for  the  two  groups  separately. 


36 


UNCLASSIFIED 


UNCLASSIFIED 


DSTO-TR-2950 


Appendix  D:  Predictors  of  live-fire  performance 


D.l.  Correlation  between  predictor  variables  and  live-fire  LF6  scores 

In  terms  of  assessing  readiness  for  live-fire  qualification,  the  most  useful  predictor  would 
be  the  scores  for  the  final  LF6  practice  in  the  simulator.  The  correlation  between  the  LF6 
scores  for  the  third  attempt  in  the  simulator  and  the  scores  for  the  first,  second  and  third 
qualification  attempts  were  r  =  0.06,  r  =  0.48  and  r  =  0.16,  respectively.  The  corresponding 
significance  values  were  p  =  0.72,  p  =  0.003  and  p  =  0.36,  respectively.  These  levels  of 
correlation  are  too  low  to  allow  useful  prediction  of  live-fire  performance.  Similar  values 
for  the  correlation  coefficients  have  been  found  in  previous  studies  comparing 
marksmanship  performance  in  the  two  environments  [3,  8, 10-11]. 

There  was  considerable  variation  in  the  correlation  between  the  scores  for  the  final 
simulator  practice  and  the  first,  second  and  third  qualification  attempts.  One  possible 
reason  for  this  is  firer  variance;  a  single  practice  may  not  be  an  accurate  reflection  of  firer 
ability.  In  order  to  reduce  the  possible  effect  of  firer  variance,  the  mean  scores  over  three 
attempts  in  the  simulator  and  on  the  live-fire  range  were  used.  The  correlation  between 
mean  LF6  live-fire  and  simulator  scores  for  all  participants  was  moderate  in  size  (r  =  0.40), 
and  was  statistically  significant  (p  =  0.02) 31 .  This  level  of  correlation  is  also  too  low  to 
provide  useful  prediction  of  live-fire  performance. 

Given  this  outcome,  additional  predictors  of  live-fire  performance  were  investigated  using 
multiple  linear  regression  (see  Section  2.5.3).  The  dependent  variables  were  the  LF6  scores 
for  first  and  second  qualification  attempts.  The  outcomes  from  the  regression  are 
summarised  below  in  Table  9.  The  use  of  a  multiple  regression  model  results  in  larger 
correlation  coefficients,  which  indicated  that  this  set  of  variables  is  a  reasonable  predictor 
of  live  fire  performance.  However,  it  is  still  not  possible  to  select  a  cut-off  value  for  the 
independent  variable  which  results  in  reliable  prediction  of  qualification.  Furthermore, 
multiple  linear  regression  is  not  a  very  practical  tool  for  training  staff  to  assess  readiness 
for  live-fire  qualification. 


Table  9.  Predictors  of  live-fire  performance  for  multiple  linear  regression 


Dependent  variable 

Predictor  variables 

r 

P 

LF6  scores  (1st  qualification 
attempt) 

number  of  times  fired  live  (past 

12  months),  marksmanship  skill 
level 

0.66 

<  0.001 

LF6  scores  (2nd  qualification 
attempt) 

mean  simulator  scores  (over  3 
attempts),  marksmanship  skill  level 

0.73 

<  0.001 

31  Correlations  between  live-fire  and  simulator  serial  scores  were  not  conducted  due  to  the  narrow 
range  of  scores  observed  for  some  serials. 
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D.2.  Predictors  of  live-fire  qualification 

The  linear  regression  model  is  a  predictor  of  live-fire  performance,  where  the  dependent 
variable  is  the  live-fire  LF6  scores  (i.e.  effectively  a  continuous  variable).  A  more  direct 
measure  of  readiness  for  qualification  would  be  to  predict  the  likelihood  of  live-fire 
qualification,  where  the  dependent  variable  is  whether  or  not  the  firer  qualified  (i.e.  a 
categorical  variable).  In  this  case,  logistic  regression  can  be  used  to  predict  the  likelihood 
of  qualification;  however,  due  to  the  small  sample  size,  logistic  regression  was  not 
conducted  on  the  data.  Instead,  the  relationship  between  skill-level  (which  was  found  to 
be  the  strongest  predictor  in  the  linear  regression  model)  and  qualification  outcomes  was 
measured  qualitatively  by  inspection  of  the  data. 

Table  10  shows  the  number  of  times  each  trainee  achieved  a  qualifying  score  (i.e.  >  165 
points)  over  four  attempts  at  LF6  on  the  live-fire  range  for  each  category  of  skill  level.  It  is 
noteworthy  that  all  trainees  who  rated  their  skill  level  as  above  average  passed  at  least  two 
out  of  their  first  four  attempts  at  LF6;  none  failed.  Conversely,  for  those  trainees  who  rated 
their  marksmanship  skill  level  as  below  average,  only  one  trainee  achieved  a  pass  score32. 
This  analysis  provides  additional  evidence  for  the  utility  of  participants'  self-rating  of 
marksmanship  skill  level  in  discriminating  between  good  and  poor  performance  at  LF6  on 
the  live-fire  range.  The  implications  of  these  findings  are  discussed  in  Section  4.2  and  4.5.5. 


Table  10.  Breakdown  of  live-fire  passes  for  each  self-rated  skill  level.  Numbers  in  brackets  are 
percentages. 


Skill  level 

N 

Passed  none 
of  four 

Passed  one 
of  four 

Passed  two 
of  four 

Passed  three 
of  four 

Passed  four 
of  four 

Below 

6 

5 

0 

1 

0 

0 

Average 

(83.3) 

(0.0) 

(16.7) 

(0.0) 

(0.0) 

Average 

21 

6 

9 

2 

2 

2 

(28.6) 

(42.9) 

(9.5) 

(9.5) 

(9.5) 

Above 

9 

0 

0 

4 

1 

4 

Average 

(0.0) 

(0.0) 

(44.4) 

(11.1) 

(44.4) 

32  Significance  testing  was  not  conducted  on  this  sample  due  to  the  low  cell  count  for  some 
categories.  More  data  would  be  required  to  validate  the  utility  of  self-ratings  for  predicting  live-fire 
performance. 
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Appendix  E:  Analysis  of  2009  and  2010  study  outcomes 

The  outcomes  of  the  2009  study  (Section  1.1)  indicated  that  a  group  that  conducted  3  hours 
training  in  the  simulator  prior  to  conducting  live-fire  (experimental  group)  achieved  100% 
LF6  qualification  after  the  first  or  second  attempt;  the  corresponding  rate  for  the  group 
that  went  straight  to  the  live-fire  range  (control  group)  was  40%  [17],  For  the  current 
study,  the  experimental  group  achieved  a  pass  rate  of  50  %  after  two  attempts  at  LF6 
qualification;  the  control  group  achieved  a  pass  rate  of  28%  at  the  same  stage.  These 
differences  can  be  explained  (in  part)  by  (1)  differences  in  the  baseline  marksmanship 
ability  levels  between  trainees  in  the  two  studies,  (2)  the  use  of  pass  rates  as  a  measure  of 
effectiveness  and  (3)  small  sample  sizes. 

For  any  population  where  the  probability  distribution  for  trainee  performance  follows  a 
normal  distribution  (or  any  similar  symmetrical  distribution),  the  impact  of  training  (i.e. 
an  increase  in  mean  performance  scores)  on  pass  rates  is  not  linear.  Rather,  it  will  be 
greatest  where  the  mean  score  of  the  population  is  equal  to  the  qualification  score,  and 
lowest  where  the  mean  score  is  much  lower  or  higher  than  the  qualification  score.  This  is 
intuitive;  if  the  majority  of  trainees  are  well  below  the  qualification  standard,  even  a 
moderate  increase  in  performance  scores  may  not  lift  many  trainees  above  the 
qualification  standard;  hence  training  will  have  little  impact  on  pass  rates.  Similarly,  if  the 
majority  of  trainees  are  already  above  the  passing  standard,  their  level  of  performance  is 
unlikely  to  increase  much  after  training.  In  this  case,  training  will  also  have  a  minimal 
impact  on  pass  rates  as  the  majority  would  already  qualify  without  training. 

In  the  current  study,  prior  to  attempting  LF6  qualification,  trainees  in  the  control  group 
were  well  below  the  required  standard  of  165  points  such  that  only  22%  passed  after  one 
attempt  and  28%  after  two  attempts.  In  contrast,  in  the  2009  trial,  40%  of  trainees  in  the 
control  group  passed  first  or  second  attempt  consistent  with  their  standard  being  closer  to 
the  qualification  score.  Assuming  both  groups  in  the  2009  study  were  of  similar  standard, 
this  means  that  even  if  the  additional  simulator  training  led  to  a  small  increase  in  LF6 
scores,  this  could  still  have  a  marked  effect  on  pass  rates  for  the  experimental  group.  This 
is  illustrated  in  Figure  4,  which  shows  the  qualification  rates  for  the  current  study  after 
adding  5  points  to  the  LF6  qualification  scores  for  both  groups  in  order  to  simulate  the 
effect  of  raising  the  baseline  marksmanship  ability  level.  This  has  no  effect  on  the 
difference  between  the  mean  scores  for  the  two  groups,  however,  there  is  a  marked  effect 
on  the  pass  rate  and  the  results  appear  to  be  closer  to  those  from  the  2009  study.  An 
additional  effect  of  this  data  manipulation  is  that  the  differences  in  qualification  rates 
between  the  two  groups  would  be  highly  significant  after  the  first  and  second  attempts 
(first  attempt;  %2  =  8.04,  p  =  0.005  (df  =  1),  second  attempt;  y}  =  8.42,  p  =  0.004  (df  =  1).  For 
subsequent  attempts,  there  would  be  no  statistically  significant  differences  in  pass  rates. 
The  data  for  the  two  groups  in  the  2009  study  would  be  required  in  order  to  explore  this 
idea  further;  unfortunately,  the  data  is  not  available. 
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Figure  4.  Pass  rates  for  the  current  study  after  LF6  scores  have  been  modified  by  adding  5  points. 


Pass  rates  are  also  highly  sensitive  to  differences  in  marksmanship  levels  between  control 
and  experimental  groups  due  to  the  small  sample  sizes  involved.  With  a  sample  size  of  18 
in  each  group,  taking  a  firer  below  the  passing  standard  from  one  group  and  swapping 
them  with  a  firer  above  the  passing  standard  in  the  other  group  would  result  in  an  11% 
difference  in  pass  rates  between  the  two  groups.  The  corresponding  effect  on  the 
difference  in  mean  scores  between  the  groups  would  be  far  less.  In  the  above  example,  if 
the  first  firer  was  20  points  below  the  passing  standard  and  the  second  firer  20  points 
above  the  passing  standard,  the  net  effect  on  the  difference  between  the  mean  scores  of  the 
two  groups  would  be  40/18  =  2.2  points,  which  is  negligible. 


40 


UNCLASSIFIED 


UNCLASSIFIED 

DSTO-TR-2950 

Appendix  F:  Effect  of  live-fire  on  simulator 
performance  (reverse  transfer  of  training) 

The  first  measure  used  to  determine  the  effect  of  live  LF6  on  subsequent  simulator 
performance  is  the  difference  in  pass  rates  at  each  attempt  at  LF6  in  the  simulator.  The 
pass  rates  after  each  LF6  practice  in  the  simulator  are  shown  in  Figure  533.  After  three 
attempts,  50.0%  of  all  trainees  had  passed  LF6.  Of  these,  only  one  trainee  passed  all  three 
attempts.  The  pass  rates  for  the  control  group  (after  conducting  three  LF6  on  the  live-fire 
range)  are  slightly  higher  than  those  of  the  experimental  group  at  each  attempt  at  LF6. 
However,  none  of  these  differences  were  statistically  significant  (y2  =  1.6,  p  =  0.21  (df=l), 
first  attempt;  y}  =  3.6,  p  =  0.06  (df=l),  second  attempt;  yf  =  0.9,  p  =  0.34  (df=l),  third 
attempt).  Consequently,  the  three  LF6  practices  conducted  by  the  control  group  on  the 
live-fire  range  had  no  measurable  impact  on  pass  rates  in  the  simulator. 

60  - 1 
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Number  of  attempts  at  LF6  in  WTSS 

Figure  5.  Pass  rates  for  LF6  based  on  simulator  scores 

The  second  measure  used  to  determine  the  effect  of  live  LF6  on  subsequent  simulator 
performance  is  the  difference  in  mean  LF6  scores  from  the  three  LF6  simulator  practices 
for  both  groups.  The  mean  scores  for  all  participants,  as  well  as  those  for  each  group  in  the 
simulator  are  given  in  Table  11.  Mean  scores  are  below  the  pass  mark  of  165  points  for 
both  groups.  The  mean  score  for  the  control  group  is  6.7  points  lower  than  that  of  the 
experimental  group;  this  is  not  statistically  significant  (t  =  0.80,  p  =  0.44).  The  lack  of  a 
significant  difference  between  the  scores  for  the  two  groups  suggests  that  the  three  LF6 
practices  conducted  by  the  control  group  on  the  live-fire  range  had  no  significant  benefit 
on  their  subsequent  performance  in  the  simulator.  That  is,  there  is  no  detectable  transfer  of 
training  from  live-fire  to  the  simulator  for  the  LF6  practice.  This  is  consistent  with  the 
trends  observed  with  the  pass  rates  for  the  two  groups. 


33  Strictly  speaking,  during  the  course,  trainees  can  only  qualify  on  LF6  on  the  live  range.  The  pass 
rate  referred  to  in  this  instance  indicates  scores  of  165  points  or  greater. 
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Table  11.  Mean  LF6  scores  in  simulator.  Numbers  in  brackets  are  one  standard  deviation. 


Group 

First 

attempt 

Second 

attempt 

Third 

attempt 

Change 
(first  to 
third 
attempt) 

Average 
of  three 
attempts 

Experimental 

130.6 

148.8 

143.5 

12.9 

140.8 

group 

(26.9) 

(18.9) 

(23.1) 

(16.2) 

Control 

130.3 

141.1 

131.1 

0.8 

134.2 

group 

(27.3) 

(42.7) 

(37.2) 

(31.6) 

All 

130.4 

144.9 

137.1 

6.7 

137.5 

participants 

(26.7) 

(33.1) 

(31.3) 

(25.0) 

The  mean  scores  for  each  LF6  serial  in  the  simulator  are  given  in  Table  12  for  each  group 
and  for  all  participants.  As  was  the  case  for  live-fire  scores,  participants  performed  better 
on  the  100  m  serials  (6,  7  and  8)  than  on  those  serials  at  ranges  of  200  m  (3,  4  and  5)  and 
300  m  (1  and  2).  There  were  no  statistically  significant  differences  between  the  scores  for 
the  experimental  and  control  groups  for  any  of  the  different  serials.  That  is,  there  is  no 
detectable  transfer  of  training  from  live-fire  to  the  simulator  for  any  of  the  LF6  serials. 


Table  12.  Mean  LF6  scores  in  simulator  for  each  serial.  Numbers  in  brackets  are  one  standard 
deviation.  HPS  is  30  points  (Serials  1-7)  and  25  points  (Serial  8). 


Group 

Serial 

1 

2 

3 

4 

5 

6 

7 

8 

deliberate. 

snap, 

rapid, 

snap, 

rapid. 

snap, 

snap, 

moving 

300  m 

300  m 

200  m 

200  m 

200  m 

100  m 

100  m 

target 

Experimental 

15.7 

13.0 

15.6 

12.7 

13.3 

25.2 

20.6 

21.9 

group 

(3.3) 

(4.9) 

(3.8) 

(3.8) 

(4.4) 

(3.2) 

(3.7) 

(2.5) 

Control  group 

12.6 

11.3 

13.7 

13.9 

13.6 

25.0 

22.2 

21.9 

(5.9) 

(5.0) 

(6.6) 

(6.6) 

(6.4) 

(6.0) 

(3.8) 

(2.5) 

All 

14.2 

12.2 

14.6 

13.3 

13.5 

25.1 

21.4 

21.9 

Participants 

(5.0) 

(5.0) 

(5.4) 

(5.3) 

(5.4) 

(4.7) 

(3.8) 

(2.5) 
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Appendix  G:  Trainee  observations 

The  following  sections  summarise  the  comments  made  by  the  participants  (and  training 
staff)  in  response  to  the  questions  asked  at  the  conclusion  of  the  study. 

G.l.  Comparison  of  simulator  and  live-fire  experiences 

There  was  consensus  among  the  participants  that  the  simulator  provided  a  useful 
environment  for  rehearsing  the  LF6  procedures  prior  to  firing  the  same  practice  on  the 
live-fire  range.  However,  a  number  of  participants  commented  that  they  had  difficulty 
seeing  the  LF6  targets  in  the  simulator,  particularly  when  targets  were  at  longer  ranges 
(e.g.  200  m  and  300  m).  This  comment  is  consistent  with  the  performance  data  (Appendix 
C)  that  showed  that  firers  performed  worse  in  the  simulator  than  on  the  live-fire  range  for 
targets  at  longer  ranges.  It  was  suggested  by  one  participant  that  the  contrast  and 
resolution  of  the  background  and  target  imagery  in  the  simulator  for  the  LF6  practices 
could  be  modified  to  improve  the  ability  to  visualise  targets.  The  participants  also  noted 
that  the  recoil  force  of  the  simulated  M4  weapon  is  lower  than  the  recoil  force  of  the  real 
M4  weapon.  It  was  highlighted  by  the  participants  that  one  implication  of  having  different 
recoil  profiles  is  the  impact  on  the  time  to  re-establish  a  point-of-aim  and  re-engage 
targets,  which  could  affect  subsequent  LF6  performance  on  the  live-fire  range  after  firing 
in  the  simulator  (given  that  some  serials  have  targets  with  brief  exposure  times  (e.g.  three 
seconds). 

Taken  together,  these  comments  highlight  limitations  in  the  simulator,  which  are  likely  to 
impact  on  the  training  effectiveness  of  these  devices.  This  is  consistent  with  the  overall 
study  findings,  which  suggest  that  the  simulator  is  not  an  effective  substitute  for  live 
training. 


G.2.  LF6  serials 

There  were  no  major  concerns  expressed  by  the  participants  regarding  their  performance 
on  specific  LF6  serials  they  conducted  in  the  simulator  and  live-fire  range  (beyond  the 
issue  of  target  visibility  discussed  above).  However,  three  minor  points  of  interest  were 
noted.  Firstly,  one  participant  stated  that  the  moving  targets  in  Serial  8  appeared  to  move 
quicker  on  the  live-fire  range  than  the  equivalent  ones  in  the  simulator;  whether  or  not  this 
perception  is  correct  would  need  to  be  verified  by  objective  testing. 

Secondly,  the  original  intention  for  the  study  methodology  was  to  have  the  six  LF6 
practices  be  conducted  over  two  days  to  facilitate  skill  acquisition  and  reduce  the 
likelihood  of  fatigue  affecting  the  trainees'  performance.  Due  to  the  availability  of  the 
simulator  at  the  time,  all  LF6  practices  were  conducted  in  one  day.  The  participants  were 
asked  whether  conducting  six  LF6  practices  was  "too  much  in  one  day"  and  the  responses 
were  equally  split  between  yes  and  no.  It  may  be  worth  noting  that  the  amount  of  firing 
conducted  during  the  study  was  not  significantly  different  from  that  typically  conducted 
on  the  training  course.  Furthermore,  there  was  a  break  between  each  set  of  three  LF6 
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practices,  as  well  as  35-40  minutes  between  each  detail.  During  these  breaks,  participants 
were  not  conducting  any  strenuous  physical  or  mental  activities.  On  this  basis,  fatigue  is 
not  likely  to  have  affected  the  study  outcomes.  However,  the  level  of  engagement  by 
participants  in  the  control  group  when  firing  LF6  in  the  simulator  may  not  have  been 
particularly  high,  given  that  they  had  already  fired  3  LF6  practices  on  the  live-fire  range, 
and  the  fact  that  the  simulator  practices  did  not  count  towards  LF6  qualification. 

Thirdly,  participants  indicated  that  they  would  have  preferred  to  receive  specific  feedback 
about  their  LF6  performance  on  the  live-fire  range.  However,  when  LF6  is  conducted  as  a 
test,  coaching  is  not  permitted  [22],  Therefore  any  LF6  practice  in  which  coaching  was 
provided  could  not  count  towards  qualification.  In  addition,  as  mentioned  previously 
(Section  2.3),  it  was  agreed  between  the  instructional  staff  and  study  team  that  no  coaching 
would  be  provided  to  participants  during  the  study.  The  issue  of  marksmanship  coaching 
is  discussed  in  the  Section  4.5.4 


G.3.  M4  weapon  and  iron  sights 

There  was  agreement  from  all  participants  that  aiming  at  and  hitting  targets  with  the  M4 
iron  sights  is  a  more  difficult  task  compared  to  other  weapons  with  optical  sights  (e.g.  F88 
Steyr).  In  particular,  when  aiming  at  targets  with  the  Steyr,  the  target  is  designed  to  fill  the 
reticule  of  the  optical  sight  (when  aiming  at  an  LF1  target  at  100  m),  whereas  the  technique 
for  acquiring  a  target  with  the  iron  sight  is  quite  different,  as  discussed  in  Section  4.5.3. 


44 


UNCLASSIFIED 


Page  classification:  UNCLASSIFIED 


DEFENCE  SCIENCE  AND  TECHNOLOGY  ORGANISATION 

DOCUMENT  CONTROL  DATA  i-  privacy  marking/ caveat  (of  document) 

2.  TITLE  3.  SECURITY  CLASSIFICATION  (FOR  UNCLASSIFIED  REPORTS 

THAT  ARE  LIMITED  RELEASE  USE  (L)  NEXT  TO  DOCUMENT 
Evaluation  of  the  Effectiveness  of  Simulation  for  M4  Marksmanship  CLASSIFICATION) 

Training 

Document  (U) 

Title  (U) 

Abstract  (U) 

4.  AUTHOR(S)  5.  CORPORATE  AUTHOR 

Ashley  Stephens  and  Philip  Temby  DSTO  Defence  Science  and  Technology  Organisation 

506  Lorimer  St 

Fishermans  Bend  Victoria  3207  Australia 


6a.  DSTO  NUMBER  6b.  AR  NUMBER  6c.  TYPE  OF  REPORT  7.  DOCUMENT  DATE 

DSTO-TR-2950  AR-015-901  Technical  Report  Feburary  2014 


8.  FILE  NUMBER 

9.  TASK  NUMBER 

10.  TASK  SPONSOR 

11.  NO.  OF  PAGES 

12.  NO.  OF  REFERENCES 

2013/1081870/1 

ARM  07/163 

DGSOC 

44 

28 

13.  DSTO  Publications  Repository  14.  RELEASE  AUTHORITY 


http://dspace.dsto.defence.gov.au/dspace/  Chief,  Land  Division 

15.  SECONDARY  RELEASE  STATEMENT  OF  THIS  DOCUMENT 

Approved  for  public  release. 

OVERSEAS  ENQUIRIES  OUTSIDE  STATED  LIMITATIONS  SHOULD  BE  REFERRED  THROUGH  DOCUMENT  EXCHANGE,  PO  BOX  1500,  EDINBURGH,  SA  5111 _ 

16.  DELIBERATE  ANNOUNCEMENT 

No  Limitations 

17.  CITATION  IN  OTHER  DOCUMENTS  Yes 

18.  DSTO  RESEARCH  LIBRARY  THESAURUS  http://web-vic.dsto.defence.gov.au/workareas/library/ resources/  dsto_thesaurus.shtml 
Training,  training  evaluation,  simulation,  marksmanship 

19.  ABSTRACT 

Simulation  is  widely  used  within  the  Australian  Army  and  other  defence  forces  for  marksmanship  training.  However,  there  is  a  lack  of 
data  that  quantifies  the  training  effectiveness  of  these  devices,  and  informs  the  appropriate  mix  of  live-fire  and  simulation.  This  report 
documents  a  study  which  measured  the  effect  of  additional  simulator  training  on  a  basic  live-fire  qualification  task  with  the  M4 
weapon.  The  study  was  conducted  with  thirty-six  Australian  Defence  Force  personnel  enrolled  in  an  Army  training  course.  Following 
initial  marksmanship  training,  one  group  (n=18)  completed  additional  simulator  training  while  the  other  group  (n=18)  proceeded 
straight  to  live-fire  qualification.  The  group  that  received  additional  training  in  the  simulator  achieved  slightly  faster  live-fire 
qualification;  however  trainees  in  both  groups  required  numerous  attempts  to  pass.  The  findings  show  that  the  simulator  training  had  a 
small,  positive  benefit,  but  that  the  overall  training  method  was  not  particularly  satisfactory.  The  report  outlines  reasons  for  these 
findings  and  provides  recommendations  for  future  studies  that  could  explore  more  effective  training  methodologies. 

Page  classification:  UNCLASSIFIED 


